1 to 25 of 71 Apache Airflow Jobs in London

complex legacy data pipelines into production-grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high-quality testing frameworks covering performance, data quality and reliability Leveraging Agentic AI development tools to improve … consumed We're looking for someone with experience in Staff, Principal or Lead Data Engineering roles Python and expert-level PySpark AWS Glue Apache Spark Apache Airflow Cloud-native Data Engineering architectures Large-scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing, validation ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

warehousing concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical ...

delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality checks into pipelines. ...

Engineering with a strong focus on ETL/ELT development. Strong proficiency in SQL and Python. Hands-on experience with DBT, Git, Terraform, IAM, Airflow/Composer, and Docker. Cloud platform experience, preferably Google Cloud Platform (BigQuery), with Azure or AWS also considered. Experience building and maintaining modern data ...

analytics infrastructure. Set engineering standards, tooling decisions, and technical direction across the data stack (Python, SQL, DBT, AWS: S3, Glue, EMR, Lambda, Aurora, Airflow, Terraform). Ensure data availability, integrity, and performance as the platform scales globally. Drive best practices in data quality, automated testing, anomaly detection ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

platforms (e.g. Snowflake, BigQuery, AWS Glue, Azure Data Factory, Data bricks, DataOps) Proficiency in using CI/CD tools and data orchestration tools (eg. Airflow/Luigi) Proven track record of independently design, develop and delivering cost effective data & technology infrastructure solutions Benefits Company bonus Enhanced Pension Wider benefits ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

roles Strong Python; experience with distributed data systems Relational DB experience (PostgreSQL preferred) NoSQLExposure to vector databases (Qdrant, Milvus, pg vector) Bonus: Spark/Airflow/Kafka, cloud (GCP/AWS/Azure), AI/ML platform experience, early-stage startup bac kground Why Join Build the data foundations ...

environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high … commercial experience developing data engineering solutions using Python . Extensive hands-on experience with Databricks , including Workflows, Notebooks and Delta Lake. Strong experience with Apache Spark/PySpark . Proven experience building scalable ETL/ELT pipelines. Strong SQL and data modelling skills. Experience designing modern Lakehouse architectures. Experience ...

reviews to maintain high engineering standards. As a Data Engineer you will have the following key skills Cloud: GCP Warehousing: BigQuery/Snowflake Orchestration: Airflow Backend: Python, SQL, Shell Languages: Scala/Java DevOps: Terraform, Docker, Kubernetes Observability: Monte Carlo BI: Looker AI/ML: MLOps London | 12 Months ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

planning, and UAT support. Experience with cloud‐based data platforms, particularly Snowflake. Understanding of ETL/ELT frameworks and tools such as dbt and Apache Airflow. Knowledge of data governance, metadata management, and data lineage concepts. Experience supporting Business Intelligence and reporting solutions. Familiarity with modern analytics engineering practices. ...

ownership. Experience designing and enforcing data contracts and quality SLAs in distributed data ecosystems. Proficiency in Python or Java, and experience building pipelines with Airflow, Spark, or Kafka. Strong understanding of data governance principles, privacy regulations (such as the GDPR, CCPA), and secure data handling practices. Familiarity with metadata ...

approaches, fit to the use case Build and maintain transformation layers in dbt (or equivalent), with appropriate testing, documentation and lineage Operate orchestration tooling (Airflow, Dagster, Prefect or equivalent) and the workflows around it Manage cloud data warehouses and lakehouses (Snowflake, BigQuery, Redshift, Synapse, Databricks) Implement data quality, testing … Python skills (or Scala/Java where relevant) for pipeline development and tooling Hands-on experience with dbt or equivalent transformation tooling Orchestration experience (Airflow, Dagster, Prefect or equivalent) Production experience with at least one major cloud warehouse/lakehouse (Snowflake, BigQuery, Redshift, Synapse, Databricks) Cloud platform experience ...

Proven experience in a Lead/Senior Data Engineering role Strong handson experience with Snowflake Expertise in ETL/ELT and orchestration tools (e.g. Airflow, dbt, Matillion) Strong SQL and Python skills Experience building scalable data platforms in cloud environments Previous experience leading or mentoring engineers Strong communication ...

Snowflake Toolkit: Hands-on with Snowpipe, Streams, Tasks, Dynamic Tables, and Stored Procedures. Nice to Have: Exposure to DBT, Snowpark, Snowflake AI/ML, Airflow, and Snowflake administration (RBAC, warehouse sizing). Mindset: Strong problem-solving abilities with an ownership mindset for end-to-end delivery. This ...

Strong cloud data platform expertise, including Google Cloud Platform (GCP). Hands-on experience with BigQuery, Dataflow, Datastream, Data Fusion, Cloud Functions, Dataproc, and Airflow/Composer. Advanced SQL and Python skills. Expertise in data lakes, data warehouses, and modern data architectures. Strong understanding of data governance, lineage ...

orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic and transparency of underlying code Owned a project across a whole ...

systems Excellent communication skills, able to explain complex technical systems to commercial, operations and senior leadership teams. Experience with Google Cloud, Docker, Kafka, Spark, Airflow, ElasticSearch, ClickHouse or similar production infrastructure Experience owning product roadmaps, release governance or cross-functional delivery across distributed teams and time zones. What ...

technologies and practices such as Docker, Terraform, CI/CD pipelines and infrastructure‐as‐code. Familiarity with MLOps tools and practices such as MLflow, Airflow, model monitoring, API monitoring, data validation, data drift detection, autoscaling and access management. Strong Python experience, with helpful knowledge of Spark or PySpark ...

similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture and Lakehouse concepts Advanced SQL and distributed … other highly regulated enterprise environments would be highly advantageous Desirable experience includes: Databricks certifications Kafka or Structured Streaming CI/CD and DevOps practices Airflow or Databricks Workflows Delta Lake optimisation techniques Docker and Kubernetes Exposure to machine learning pipelines Job Offer Competitive day rate ...

best practices and coding standards. Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using Apache Airflow, Spark, SQL, Python, and ETL tools. CI/CD & DevOps Design and implement CI/CD pipelines for data platforms. Automate code ...

supporting modern cloud-based data platforms. Understanding of data modelling and data transformation principles. Ability to work within collaborative engineering teams. Desirable Skills Airflow Databricks AI-focused development experience Semantic layer experience Exposure to Azure-based environments and integrations. Reasonable Adjustments: Respect and equality are core values to us. ...