1 to 25 of 81 Apache Airflow Jobs in London

execute. You have deep experience designing, building, and operating production data pipelines, including owning reliability, monitoring, and failure recovery for orchestration systems such as Apache Airflow (or equivalent). You are fluent in modern data and cloud ecosystems, with hands-on experience (or the ability to ramp quickly … days a week in office Lean principles and Agile development practices Continuous deployment across all microservices Big Data ecosystem and SQL Databases (Apache Airflow, Iceberg, Spark/Thrift, PostgreSQL, Azure Hyperscale) Python as a primary backend language High-level frameworks (Flask, SQLAlchemy, Alembic, Pytest, Socket.IO) Amazon Web Services ...

automate workflows, and deliver secure, high-performance data platforms using AWS-native technologies. Key Responsibilities Develop and maintain data pipelines using Python , SQL , and Apache Airflow . Build and manage containerised data applications using Docker . Design and optimise solutions using AWS services including Redshift, OpenSearch, Lambda, Glue … data governance, security, and best practice standards. Essential Skills Active Security Clearance Strong experience with Python and SQL . Hands-on experience with Apache Airflow and Docker . Proven experience with AWS data services, including Redshift, OpenSearch, Lambda, Glue, Step Functions, and Batch . Experience with Git , GitLab ...

Description Data Engineer - GCP, BigQuery, Airflow Location: London (Hybrid)Contract: 12 MonthsRate: £350 per day PAYE (Inside IR35)Start Date: ASAP A leading technology and data-driven organisation is seeking an experienced Data Engineer to join its data engineering team on an initial 12-month contract. The successful candidate … across critical business data platforms. Key Responsibilities Build, maintain, and optimise scalable data pipelines within GCP Develop and manage workflows using Cloud Composer and Apache Airflow Design and support data solutions using BigQuery Implement data quality checks and monitoring frameworks Improve observability and operational performance of data platforms ...

Data Engineer - GCP, BigQuery, Airflow Location: London (Hybrid) Contract: 12 Months Rate: £350 per day PAYE (Inside IR35) Start Date: ASAP A leading technology and data-driven organisation is seeking an experienced Data Engineer to join its data engineering team on an initial 12-month contract. The successful candidate … across critical business data platforms. Key Responsibilities Build, maintain, and optimise scalable data pipelines within GCP Develop and manage workflows using Cloud Composer and Apache Airflow Design and support data solutions using BigQuery Implement data quality checks and monitoring frameworks Improve observability and operational performance of data platforms ...

warehousing concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical ...

Lead the evolution of the enterprise data estate, leveraging modern data platforms such as Databricks, Snowflake, Azure Synapse, and BigQuery. Oversee data engineering using Apache Airflow, dbt, and Prefect, ensuring data pipelines are performant, governed, and aligned with enterprise metadata standards (Collibra, Alation, Microsoft Purview). Drive ...

delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality checks into pipelines. ...

Engineering with a strong focus on ETL/ELT development. Strong proficiency in SQL and Python. Hands-on experience with DBT, Git, Terraform, IAM, Airflow/Composer, and Docker. Cloud platform experience, preferably Google Cloud Platform (BigQuery), with Azure or AWS also considered. Experience building and maintaining modern data ...

Integration: Sqoop, Kafka, Flume, NiFi Programming: Python, Scala, Java, Shell Scripting Databases: Oracle, SQL Server, MySQL, PostgreSQL Cloud: AWS, Azure, GCP Workflow Tools: Oozie, Airflow We are a Disability Confident Employer: Capgemini is proud to be a Disability Confident Employer (Level 2) under the UK Government’s Disability Confident ...

Integration: Sqoop, Kafka, Flume, NiFi Programming: Python, Scala, Java, Shell Scripting Databases: Oracle, SQL Server, MySQL, PostgreSQL Cloud: AWS, Azure, GCP Workflow Tools: Oozie, Airflow We are a Disability Confident Employer: Capgemini is proud to be a Disability Confident Employer (Level 2) under the UK Government’s Disability Confident ...

Integration: Sqoop, Kafka, Flume, NiFi Programming: Python, Scala, Java, Shell Scripting Databases: Oracle, SQL Server, MySQL, PostgreSQL Cloud: AWS, Azure, GCP Workflow Tools: Oozie, Airflow We are a Disability Confident Employer: Capgemini is proud to be a Disability Confident Employer (Level 2) under the UK Government’s Disability Confident ...

Integration: Sqoop, Kafka, Flume, NiFi Programming: Python, Scala, Java, Shell Scripting Databases: Oracle, SQL Server, MySQL, PostgreSQL Cloud: AWS, Azure, GCP Workflow Tools: Oozie, Airflow We are a Disability Confident Employer: Capgemini is proud to be a Disability Confident Employer (Level 2) under the UK Government’s Disability Confident ...

scripting and data engineering. Core Tech Stack: Deep proficiency with Python, Postgres SQL, Azure Databricks, AWS (S3), Git, Azure DevOps CI/CD, and Apache Airflow. Python Ecosystem: Extensive, recent hands-on experience utilizing NumPy, Pandas, Polars, BeautifulSoup, Selenium, and Requests . Data Ecosystems: Strong background in building native ...

scripting and data engineering. Core Tech Stack: Deep proficiency with Python, Postgres SQL, Azure Databricks, AWS (S3), Git, Azure DevOps CI/CD, and Apache Airflow. Python Ecosystem: Extensive, recent hands-on experience utilizing NumPy, Pandas, Polars, BeautifulSoup, Selenium, and Requests . Data Ecosystems: Strong background in building native ...

scripting and data engineering. Core Tech Stack: Deep proficiency with Python, Postgres SQL, Azure Databricks, AWS (S3), Git, Azure DevOps CI/CD, and Apache Airflow. Python Ecosystem: Extensive, recent hands-on experience utilizing NumPy, Pandas, Polars, BeautifulSoup, Selenium, and Requests . Data Ecosystems: Strong background in building native ...

scripting and data engineering. Core Tech Stack: Deep proficiency with Python, Postgres SQL, Azure Databricks, AWS (S3), Git, Azure DevOps CI/CD, and Apache Airflow. Python Ecosystem: Extensive, recent hands-on experience utilizing NumPy, Pandas, Polars, BeautifulSoup, Selenium, and Requests . Data Ecosystems: Strong background in building native ...

Azure (experience with AWS or GCP a plus). Strong technical proficiency with the modern data engineering stack, including tools such as dbt, Coalesce, Airflow, Fivetran, and Python. Familiarity with DataOps practices, including CI/CD and infrastructure-as-code principles. Demonstrated success managing distributed teams across multiple geographies. ...

architecture and end-to-end system design Proven Experience with big data platforms at scale Deep knowledge of open-source data ecosystems, such as: Apache Spark Trino Airflow Apache Superset Cloud & Infrastructure Experience across cloud and on-premise environments Familiarity with AWS, GCP, and/or Azure ...

tools such as dbt, Matillion, Informatica, Fivetran, or similar. Strong Python programming skills for automation and data engineering. Experience with orchestration tools such as Apache Airflow or similar. Experience implementing CI/CD pipelines using Git, Azure DevOps, GitHub Actions, or Jenkins. Good understanding of dimensional modelling, data ...

data platforms (e.g. Snowflake) • Data Modelling Skills: Proven ability to design scalable, well-structured and reusable data models • Tooling Experience: Experience with dbt, Python, Airflow or similar modern data stack tools • DataOps Practices: Familiarity with git, CI/CD and testing frameworks for data pipelines • Data Quality Focus: Strong ...

cost optimization, Horizon Catalog, Cortex). dbt (modular analytics engineering, transformations, testing). AWS Python proficiency as a data engineer and Orchestration tools (e.g. Airflow) Data and Dimensional modelling (star schema, Kimball/Inmon methodologies, OBT). AI - copilots, agentic AI, MCP Servers Strong understanding of DataOps, CI/ ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...