26 to 50 of 251 Apache Airflow Jobs

from various structured and unstructured sources (databases, APIs, streaming platforms) into BigQuery and Cloud Storage Implementing data ingestion and real-time processing using Dataflow (Apache Beam) and Pub/Sub for batch and streaming workflows Developing SQL transformation workflows with Dataform , including version control, testing, and automated scheduling with … built-in quality assertions Creating efficient, cost-optimized BigQuery queries with proper partitioning, clustering, and denormalization strategies Orchestrating complex workflows using Cloud Composer (Apache Airflow) and Cloud Functions for event-driven data processing Implementing centralized data governance and metadata management using Dataplex with automated cataloging and lineage tracking ...

from various structured and unstructured sources (databases, APIs, streaming platforms) into BigQuery and Cloud Storage Implementing data ingestion and real-time processing using Dataflow (Apache Beam) and Pub/Sub for batch and streaming workflows Developing SQL transformation workflows with Dataform , including version control, testing, and automated scheduling with … built-in quality assertions Creating efficient, cost-optimized BigQuery queries with proper partitioning, clustering, and denormalization strategies Orchestrating complex workflows using Cloud Composer (Apache Airflow) and Cloud Functions for event-driven data processing Implementing centralized data governance and metadata management using Dataplex with automated cataloging and lineage tracking ...

running production pipelines end to end Excellent Python and SQL Solid grasp of data modelling, warehousing and lakehouse concepts Hands-on with orchestration (e.g. Airflow, Dagster) and distributed processing (e.g. Spark) Experience with cloud data platforms (AWS, Azure or GCP) and containerised deployment (Docker, Kubernetes) Good engineering practice: version ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

experience with cloud data platforms such as Snowflake, Databricks, Azure Data Services or equivalent AWS technologies.Experience with modern transformation and orchestration tools including dbt, Airflow, Azure Data Factory or similar.A solid understanding of statistics and machine learning techniques including regression, classification, forecasting and validation methodologies.Demonstrated experience operationalising data products ...

cloud data AWS Aurora PostgreSQL AWS Database Migration Service (DMS) Snowflake Azure Google Cloud Platform (GCP) Data integration and engineering Informatica Talend Apache Airflow Python for data processing and automation Data warehouse design and optimisation APIs and event-driven architecture REST APIs GraphQL Apache Kafka Event-driven ...

compliant solutions that align with GDPR and healthcare data governance standards Stack: Snowflake Snowflake Cortex AWS - S3, Glue, Lambda, Step Functions Python SQL dbt Airflow Terraform Docker GitHub CI/CD Requirements Must have advanced SQL and Python + experience delivering modern cloud-based data platforms Must have enterprise ...

compliant solutions that align with GDPR and healthcare data governance standards Stack: Snowflake | Snowflake Cortex | AWS - S3, Glue, Lambda, Step Functions | Python | SQL | dbt | Airflow| Terraform | Docker | GitHub CI/CD Requirements Must have advanced SQL and Python + experience delivering modern cloud-based data platforms Must have enterprise ...

/on-prem environments. The ideal candidate will have: * Experience in data engineering, ideally aerospace/defence or industrial sectors. * ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka). * Strong Python and SQL; C++, C#, or MATLAB a plus. * Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg ...

Service - integration model on hybrid/on-prem environments.Hard SkillsExperience in data engineering, ideally aerospace/defence or industrial sectors.ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka).Strong Python and SQL; C++, C#, or MATLAB a plus.Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB.Data Platform Capabilities ...

Service - integration model on hybrid/on-prem environments.Hard SkillsExperience in data engineering, ideally aerospace/defence or industrial sectors.ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka).Strong Python and SQL; C++, C#, or MATLAB a plus.Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB.Data Platform Capabilities ...

integration model on hybrid/on-prem environments. Hard SkillsExperience in data engineering, ideally aerospace/defence or industrial sectors. ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka).Strong Python and SQL; C++, C#, or MATLAB a plus. Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB.Data ...

based and agentic applications solid data engineering experience with large, complex financial datasets, plus confidence working with cloud and containerized deployments experience with Kubernetes, Apache Airflow, Microsoft Azure, Apache Spark and kdb+/q is welcome but not required knowledge of Global Markets and sales workflows ...

Terraform and Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery.Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines.Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

Desirable experience It would be beneficial if you also have experience with: MySQL Python Docker and containerisation CI/CD pipelines Terraform or CloudFormation Airflow, AWS Step Functions or other workflow orchestration tools Building APIs or packaging models in R, such as plumber or R packages SaaS or software ...

statistical & analytical skills (e.g. sampling methods), attention to detail, and a proactive empirical mindsetFamiliarity with data modelling & modern data tech stack tools such as Airflow, dbt, Kafka, Iceberg, Trino, Superset etc.Familiarity with some data visualization tools (Tableau, QlikSense, Power BI, or similar) to communicate quality insightsFamiliarity with at least ...

relational databases (e.g., PostgreSQL, DuckDB). Experience with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI).Data Science: Familiarity or experience ...

formats like Parquet. MLOps Exposure: Experience with tools and frameworks for training, tracking, deploying, and monitoring models in production (e.g., MLflow, Optuna, Prefect, or Airflow). Desirable Background in supply chain forecasting, enterprise AI, or retail analyticsM.S. or Ph.D. in Computer Science, Artificial Intelligence, Statistics, Data Science ...

Scrum or Kanban.Strong analytical and problem-solving skills, with a focus on delivering high-quality solutions.Experience with modern data engineering tools such as Airflow, dbt, Dataform, or similar technologies.Excellent communication and stakeholder management skills, with the ability to explain technical concepts to non-technical audiences.Unfortunately we are unable ...

such as SageMaker, Vertex AI or Azure MLData engineering and MLOpsExperience with data warehouses such as Snowflake, BigQuery or RedshiftWorkflow orchestration using tools like Airflow or DagsterExperience with MLOps tools such as MLflow, Weights and Biases or similarAwareness of data and model drift, and how to monitor and respond ...

effective working with a distributed team (London, Boston, and New York) and directly with researchers and vendors. Desirable QualificationsExperience with pipelines and orchestration (e.g., Airflow, Prefect, Dagster, or similar) - strongly preferred. Practical experience applying AI tools to data work (coding assistance, investigation, anomaly detection).Experience supporting production data pipelines ...

with using and integrating BI tools such as QuickSight, Looker, Power BI, Tableau Business Intelligence (BI) knowledge/experience Knowledge on tools like Terraform, Airflow, DBT and Harness Modern data engineering practises The environment is that of Facebook or Google, relaxed open with time to think and make ...

skills Experience building data pipelines and integrating data from multiple sources Exposure to technologies such as Power BI, Snowflake, SQL Server, PostgreSQL, Spark, Kafka, Airflow, Docker, and Kubernetes Experience within engineering, manufacturing, industrial, or other complex regulated environments would be advantageous Excellent stakeholder management and problem-solving abilities What ...

pipelines end to end. Modern data stack Experience with modern ELT, transformation and orchestration technologies. Our environment includes technologies such as dbt, Dagster/Airflow and Snowflake, although experience with comparable technologies is equally relevant. Cloud infrastructure Experience designing and implementing secure, scalable solutions on a major cloud platform. ...