26 to 50 of 281 Apache Airflow Jobs in London

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

software engineering experience using Python and/or other modern programming languages. Experience developing and orchestrating data pipelines using technologies such as DBT, Apache Spark, Airflow, Dagster, Kafka, Flink, or equivalent tooling. Solid understanding of data lakehouse, warehouse, and medallion architecture patterns. Experience applying DevOps and DataOps best ...

running production pipelines end to end Excellent Python and SQL Solid grasp of data modelling, warehousing and lakehouse concepts Hands-on with orchestration (e.g. Airflow, Dagster) and distributed processing (e.g. Spark) Experience with cloud data platforms (AWS, Azure or GCP) and containerised deployment (Docker, Kubernetes) Good engineering practice: version ...

evaluation. Ideal candidates have strong SQL and Python skills, experience with ETL/ELT, orchestration, streaming and data warehousing, and familiarity with Spark, Kafka, Airflow, dbt, Snowflake, BigQuery or Databricks on AWS, GCP or Azure. #J-18808-Ljbffr ...

core stack of SQL and Python, with experience across ETL and ELT, orchestration, streaming and data warehousing, using tools such as Spark, Kafka, Airflow, dbt, Snowflake, BigQuery or Databricks on AWS, GCP or Azure. Some roles focus on hands‐on delivery, others on documenting and structuring that work. Commitment ...

compliant solutions that align with GDPR and healthcare data governance standards Stack: Snowflake | Snowflake Cortex | AWS - S3, Glue, Lambda, Step Functions | Python | SQL | dbt | Airflow| Terraform | Docker | GitHub CI/CD Requirements Must have advanced SQL and Python + experience delivering modern cloud-based data platforms Must have enterprise ...

compliant solutions that align with GDPR and healthcare data governance standards Stack: Snowflake Snowflake Cortex AWS - S3, Glue, Lambda, Step Functions Python SQL dbt Airflow Terraform Docker GitHub CI/CD Requirements Must have advanced SQL and Python + experience delivering modern cloud-based data platforms Must have enterprise ...

compliant solutions that align with GDPR and healthcare data governance standards Stack: Snowflake | Snowflake Cortex | AWS - S3, Glue, Lambda, Step Functions | Python | SQL | dbt | Airflow| Terraform | Docker | GitHub CI/CD Requirements Must have advanced SQL and Python + experience delivering modern cloud-based data platforms Must have enterprise ...

Hadoop, Flink) and cloud platforms (AWS, GCP, or Azure). Familiarity with data warehouse technologies (e.g., Snowflake, Redshift, BigQuery) and workflow orchestration tools (e.g., Airflow, Luigi). Strong knowledge of data security, compliance, and governance frameworks. Effective communication in English. Preferred Qualifications Experience in e‐commerce or consumer‐facing ...

cloud data warehouses (e.g. BigQuery, Snowflake, Redshift, ADW), including performance, cost, and data layout considerations. Proficiency with data integration and orchestration tools (e.g. Fivetran, Airflow, managed ELT platforms), and an understanding of how ingestion, transformation, and orchestration fit together. Experience building and maintaining an analytics or semantic modelling layer ...

Kubernetes Spark & distributed computing dbt Terraform Data Warehousing (we use Databricks but experience with other platforms is fine) MLOps implementation (desirable but not required) Airflow (or other orchestration tooling) Governance Things to note Just to let you know, we are dog friendly company ...

shared infrastructure and pipelines for reliability, monitoring and scale. Requirements: Production data platforms with high-volume ETL/ELT, distributed processing and orchestration (e.g. Airflow, Dagster, Prefect), as distinct from analytics pipelines. Python and SQL at a high level, plus Bash automation. Cloud infrastructure, with AWS preferred (S3, Lambda ...

skills in SQL; experience with Snowflake and DBT. Proficiency in SQL and NoSQL databases. Experience with containerization tools like Docker and orchestration tools like Airflow or Astronomer. Familiarity with cloud platforms (AWS, Azure). Experience with CI/CD pipelines and automation tools like GitHub Actions. Experience with data ...

Google Cloud Platform ecosystem, like Cloud Run, Cloud Functions, Cloud Storage, Pub/Sub, and Cloud Scheduler Data orchestration tools such as Cloud Composer (Airflow), Prefect, or equivalent Looker/Data Studio and “Conventional analytics” (aka. BQ agenta) Building or maintaining a data warehouse in a regulated or healthcare ...

Strong hands-on experience building and operating data platforms on GCP. Working experience with GCP data services such as BigQuery, Dataflow, Dataproc, Cloud Composer (Airflow), Pub/Sub, Cloud Storage, Bigtable, and Cloud SQL. Proficiency in Python and SQL, with experience in Spark or Apache Beam. Experience designing ...

data engineering, with demonstrated expertise in designing and implementing scalable, flexible modern data architectures Experience building modern data lake/lakehouse architectures (e.g., Apache Iceberg) in cloud environments Strong knowledge of AWS (S3, Redshift) or Azure data services Experience with ETL/ELT processes and data ingestion Proven expertise … using transformation tools such as dbt Experience with data pipeline orchestration tools (such as Apache Airflow) Experience with best practices applied to data (e.g., testing, CI/CD) Advantageous Experience developing dashboards and reporting solutions using data visualization tools such as Power BI or Amazon QuickSight Experience with ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

implementing enterprise-scale data solutions. The ideal candidate will possess strong expertise in Snowflake and hands‐on experience with Python, PySpark, AWS, and Airflow for building scalable, high-performance data platforms. The role demands strong technical leadership, problem‐solving capabilities, and the ability to collaborate effectively with cross‐functional ...

core stack of SQL and Python, with experience across ETL and ELT, orchestration, streaming and data warehousing, using tools such as Spark, Kafka, Airflow, dbt, Snowflake, BigQuery or Databricks on AWS, GCP or Azure. Some roles focus on hands-on delivery, others on documenting and structuring that work. Commitment ...

Redshift), and data transformation tools (such as dbt). Pipeline mastery: Practical experience building and operating batch or streaming pipelines using orchestration tools like Airflow, Prefect, or Dagster. Strong communicator: Skilled at writing clear technical specifications and explaining complex trade-offs to both technical and non-technical colleagues. Growth ...

automated testing and data quality practice. A collaborative mindset and a genuine interest in building things well. Desirable Experience with orchestration tools such as Airflow, Dagster, Prefect or Azure Data Factory. Experience working within AWS, Azure, GCP or another enterprise cloud ecosystem. Experience with complex customer, transactional, digital ...

data modelling Python, Git, Docker ETL pipelines and data platforms Relational database engines, including PostgreSQL and DuckDB Orchestration (e.g. Step Functions, Airflow) AI-accelerated development workflows and tools including Claude Code and Codex AWS cloud infrastructure, including Lambda and ECS End-to-end data pipeline testing strategy and implementation ...

maintaining scalable data platforms. Strong SQL and data modelling skills, with experience designing analytical datasets. Experience with modern data stack tools (e.g. dbt, Spark, Airflow, or similar orchestration and transformation frameworks). Experience with cloud‐based data platforms, ideally AWS. Understanding of medallion architecture or similar data layering approaches. ...

experience you need Strong experience in Data Engineering and cloud-based data platforms. Hands-on knowledge of Databricks, Spark, SQL and Python. Understanding of Airflow, Terraform and modern engineering practices. Experience building large-scale batch and real-time data pipelines. Strong problem-solving and decision-making skills. Ability ...

with modern data warehouse technologies, particularly Snowflake.* Exposure to technologies such as Azure Data Factory, Synapse Analytics, Databricks, Spark, SQL, and Python. Experience with Airflow would be advantageous.Data Engineering* Proven experience applying data engineering techniques to design and develop scalable solutions.* Ability to build and maintain robust data pipelines ...