1 to 25 of 80 Permanent Apache Airflow Jobs in London

execute. You have deep experience designing, building, and operating production data pipelines, including owning reliability, monitoring, and failure recovery for orchestration systems such as Apache Airflow (or equivalent). You are fluent in modern data and cloud ecosystems, with hands-on experience (or the ability to ramp quickly … days a week in office Lean principles and Agile development practices Continuous deployment across all microservices Big Data ecosystem and SQL Databases (Apache Airflow, Iceberg, Spark/Thrift, PostgreSQL, Azure Hyperscale) Python as a primary backend language High-level frameworks (Flask, SQLAlchemy, Alembic, Pytest, Socket.IO) Amazon Web Services ...

automate workflows, and deliver secure, high-performance data platforms using AWS-native technologies. Key Responsibilities Develop and maintain data pipelines using Python , SQL , and Apache Airflow . Build and manage containerised data applications using Docker . Design and optimise solutions using AWS services including Redshift, OpenSearch, Lambda, Glue … data governance, security, and best practice standards. Essential Skills Active Security Clearance Strong experience with Python and SQL . Hands-on experience with Apache Airflow and Docker . Proven experience with AWS data services, including Redshift, OpenSearch, Lambda, Glue, Step Functions, and Batch . Experience with Git , GitLab ...

Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability \n Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations \n Model and transform data for analytics using SQL and dbt to support business intelligence … Hudi, or similar) \n Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) \n Experience designing and operating orchestration pipelines using Airflow or similar tools \n Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems \n\n Preferred qualifications, capabilities ...

PySpark, and dbt\ NBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability\ NDevelop and operate workflow orchestration (e.G., Apache Airflow) to schedule, monitor, and manage data movement and transformations\ NModel and transform data for analytics using SQL and dbt to support business … such as Iceberg, Hudi, or similar)\ NStrong SQL skills and experience with SQL-based transformation tooling (e.G., dbt)\ NExperience designingand operating orchestration pipelines using Airflow or similar tools\ NExperience designingand building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems\ N\nPreferred qualifications, capabilities, and skills ...

NBuild and operate batch and streaming data pipelines with strong scalability, performance, and fault tolerance\ NDevelop and manage workflow orchestration using tools such as Apache Airflow to support reliable, observable, and well-scheduled data movement and transformations\ NImplement and optimize data models and warehouse structures to support analytics …/integration testing practices\ NExperience with SQL and familiarity with SQL-based workflow management tools such as dbt\ NExperience with orchestration tools such as Airflow (or similar)\ NUnderstanding of messaging/streaming systems such as Kafka or Pub/Sub (or similar)\ NFamiliarity with infrastructure-as-code (e.G., Terraform ...

operate batch and streaming data pipelines with strong scalability, performance, and fault tolerance \n Develop and manage workflow orchestration using tools such as Apache Airflow to support reliable, observable, and well-scheduled data movement and transformations \n Implement and optimize data models and warehouse structures to support analytics … testing practices \n Experience with SQL and familiarity with SQL-based workflow management tools such as dbt \n Experience with orchestration tools such as Airflow (or similar) \n Understanding of messaging/streaming systems such as Kafka or Pub/Sub (or similar) \n Familiarity with infrastructure-as-code ...

with distributed event stores and stream-processing platforms such as Kafka of Kinesis. ● Experience working with batch processing frameworks, such as DBT, Argo Workflows, Apache Airflow, etc. ● Familiarity with Docker, Kubernetes, Amazon EKS ● Familiarity with Continuous Integration with GitHub Actions ● Familiarity with Test-Driven Development ...

warehousing concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical ...

Experience working with distributed event stores and stream-processing platformssuch as Kafka of Kinesis.● Experience working with batch processing frameworks, such as DBT, ArgoWorkflows, Apache Airflow, etc.● Familiarity with Docker, Kubernetes, Amazon EKS● Familiarity with Continuous Integration with GitHub Actions● Familiarity with Test-Driven Development and XPOur commitment ...

delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality checks into pipelines. ...

Flink etc. \n Solid grasp of Kubernetes, CI/CD and Terraform. \n Working knowledge of ETL and ELT frameworks and orchestration tools like Airflow and dbt. \n Ability to provide clear input, guide, empower junior members of the team to achieve desired outcomes. \n Familiarity with machine learning ...

Engineering with a strong focus on ETL/ELT development. Strong proficiency in SQL and Python. Hands-on experience with DBT, Git, Terraform, IAM, Airflow/Composer, and Docker. Cloud platform experience, preferably Google Cloud Platform (BigQuery), with Azure or AWS also considered. Experience building and maintaining modern data ...

Integration: Sqoop, Kafka, Flume, NiFi Programming: Python, Scala, Java, Shell Scripting Databases: Oracle, SQL Server, MySQL, PostgreSQL Cloud: AWS, Azure, GCP Workflow Tools: Oozie, Airflow We are a Disability Confident Employer: Capgemini is proud to be a Disability Confident Employer (Level 2) under the UK Government’s Disability Confident ...

marketing teams to align data solutions with business needs Requirements Strong SQL skills and experience with ETL/ELT and pipeline orchestration tools (e.g., Airflow, dbt, Data Factory) Experience with cloud data platforms (e.g., Azure, AWS, or GCP) Proficiency in data modelling (dimensional, entity-relationship, event-driven) Understanding ...

scripting and data engineering. Core Tech Stack: Deep proficiency with Python, Postgres SQL, Azure Databricks, AWS (S3), Git, Azure DevOps CI/CD, and Apache Airflow. Python Ecosystem: Extensive, recent hands-on experience utilizing NumPy, Pandas, Polars, BeautifulSoup, Selenium, and Requests . Data Ecosystems: Strong background in building native ...

Data Warehousing (we use Databricks but experience with other platforms is fine) \n \n MLOps implementation (desirable but not required) \n \n Airflow (or other orchestration tooling) \n \n Governance \n\n Why else you'll love it here \n \n Wondering what the salary for this role ...

Data Warehousing (we use Databricks but experience with other platforms is fine)\ N\n MLOps implementation (desirable but not required)\ N\n Airflow (or otherorchestration tooling)\ N\n Governance\ N\nWhy else you'll love it here\ N\nWondering whatthe salary for this role is? Just ...

technologies like Spark, Flink etc.\ NSolid grasp of Kubernetes, CI/CD and Terraform.\ NWorking knowledge ofETL and ELT frameworks and orchestration tools like Airflow and dbt.\ NAbility to provide clear input, guide, empower junior members of the team to achieve desired outcomes.\ NFamiliarity with machine learning ...

Catalog, Cortex). \n dbt (modular analytics engineering, transformations, testing). \n AWS \n Python proficiency as a data engineer and Orchestration tools (e.g. Airflow) \n Data and Dimensional modelling (star schema, Kimball/Inmon methodologies, OBT). \n AI - copilots, agentic AI, MCP Servers \n Strong understanding ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

roles Strong Python; experience with distributed data systems Relational DB experience (PostgreSQL preferred) NoSQLExposure to vector databases (Qdrant, Milvus, pg vector) Bonus: Spark/Airflow/Kafka, cloud (GCP/AWS/Azure), AI/ML platform experience, early-stage startup bac kground Why Join Build the data foundations ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

approaches, fit to the use case Build and maintain transformation layers in dbt (or equivalent), with appropriate testing, documentation and lineage Operate orchestration tooling (Airflow, Dagster, Prefect or equivalent) and the workflows around it Manage cloud data warehouses and lakehouses (Snowflake, BigQuery, Redshift, Synapse, Databricks) Implement data quality, testing … Python skills (or Scala/Java where relevant) for pipeline development and tooling Hands-on experience with dbt or equivalent transformation tooling Orchestration experience (Airflow, Dagster, Prefect or equivalent) Production experience with at least one major cloud warehouse/lakehouse (Snowflake, BigQuery, Redshift, Synapse, Databricks) Cloud platform experience ...

Proven experience in a Lead/Senior Data Engineering role Strong handson experience with Snowflake Expertise in ETL/ELT and orchestration tools (e.g. Airflow, dbt, Matillion) Strong SQL and Python skills Experience building scalable data platforms in cloud environments Previous experience leading or mentoring engineers Strong communication ...