1 to 25 of 133 Apache Airflow Jobs in London

golden record creation, and distribution to downstream consumers across the firm. We operate on a modern cloud‐native stack centered on Snowflake , AWS , and Apache Airflow , and follow engineering best practices including CI/CD, code review, and automated testing. Key Responsibilities Design, build, and maintain scalable data … pipelines for Security Master, Account Master, and Counterparty Master using Python and Apache Airflow . Develop and optimize complex data transformations, stored procedures, and views in Snowflake , ensuring high performance and data quality. Own the end‐to‐end lifecycle of reference data — from source ingestion and normalization through ...

complex legacy data pipelines into production-grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high-quality testing frameworks covering performance, data quality and reliability Leveraging Agentic AI development tools to improve … consumed We're looking for someone with experience in Staff, Principal or Lead Data Engineering roles Python and expert-level PySpark AWS Glue Apache Spark Apache Airflow Cloud-native Data Engineering architectures Large-scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing, validation ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

closely with data engineers, analysts, and platform teams. \n Day to day duties \n \n Developing and maintaining data pipelines and processing jobs using Apache Spark and Python \n Orchestrating workflows with Apache Airflow \n Provisioning and managing cloud infrastructure on AWS using Terraform (Infrastructure as Code … contributing to design reviews, code reviews, and documentation \n\n Essential Skills & Experience \n \n Strong hands-on experience with Python \n Experience with Apache Spark for large-scale data processing \n Practical knowledge of AWS services (eg S3, Lambda, EC2, Glue, RDS - adjust as relevant) \n Experience writing ...

bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than … Astronomer lets businesses do more with their data. To learn more, visit www.astronomer.io. About this role: At Astronomer, we’re redefining how companies run Apache Airflow at scale. Our R&D organization is home to some of the most innovative minds in cloud infrastructure and open-source software. ...

bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than … else, and you have real opinions informed by real failures. Experience working on a SaaS/PaaS product across multiple cloud providers. Familiarity with Apache Airflow or workflow orchestration systems. #LI-Hybrid At Astronomer, we value diversity. We are an equal opportunity employer: we do not discriminate ...

software and site reliability engineering. Preferred: Experience with distributed ML frameworks such as Horovod or TensorFlow Distributed. Familiarity with data engineering pipelines such as Apache Airflow or Apache Spark. Knowledge of chaos engineering tools and compliance frameworks such as GDPR, SOC 2, or ISO 27001. Relocation benefits ...

3+ years of experience with Spark, Databricks, Snowflake, Hadoop, or other big data technologies . Nice to Have: Experience with orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Glue. Knowledge of streaming technologies such as Kafka, Kinesis, or Pub/Sub. Experience with data warehouse ...

warehousing concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical ...

agile SDLC environment, leveraging modern programming languages, DevOps, and test-driven development Exposure to Docker, Git hands on experience with pipeline orchestration tools like Apache Airflow, Autosys, Control-M – Preferred Apache Ariflow proficient at working with large and complex code bases using GitLab/Github, Fork/ ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

analytics infrastructure. Set engineering standards, tooling decisions, and technical direction across the data stack (Python, SQL, DBT, AWS: S3, Glue, EMR, Lambda, Aurora, Airflow, Terraform). Ensure data availability, integrity, and performance as the platform scales globally. Drive best practices in data quality, automated testing, anomaly detection ...

managing cloud infrastructure in AWS. Ideally will be a bonus if you have the following... Experience or knowledge of FluxCD, GitOps workflow, Appgate, MongoDB, Airflow (MWAA), Snowflake Experience in Apache Airflow (MWAA), AWS Backup and S3 versioning for business continuity. Programming experience in C#, Python and/ ...

marketing teams to align data solutions with business needs Requirements Strong SQL skills and experience with ETL/ELT and pipeline orchestration tools (e.g., Airflow, dbt, Data Factory) Experience with cloud data platforms (e.g., Azure, AWS, or GCP) Proficiency in data modelling (dimensional, entity-relationship, event-driven) Understanding ...

Strong Python and SQL skills, with experience building production-grade data pipelines • Data Platform Experience: Hands-on experience with modern data tools (e.g. Snowflake, Airflow, dbt) and cloud environments (preferably AWS) • Engineering Best Practice: Knowledge of CI/CD, testing, version control and infrastructure as code • Data Quality & Governance ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

maintaining scalable data pipelines and integrations. Developing robust ETL and data transformation processes to support analytics and reporting requirements. Working with Snowflake, DBT, Apache Airflow, AWS, SQL and Python to deliver high-quality data solutions. Collaborating with Data Architecture, DevOps and business stakeholders to ensure trusted and accessible … ideally within AWS. Data modelling, transformation and orchestration methodologies. Delivering high-quality, well-documented and thoroughly tested data solutions. Experience with Snowflake, DBT and Apache Airflow. Interest in AI and emerging technologies within the data engineering landscape. THE BENEFITS: You will receive a salary ...

implementing agentic guardrails. Data Engineering (30%): Practical experience implementing components of data pipelines, including real-time streaming tools (AWS Kinesis, Kafka), data orchestration (dbt, Airflow), and managing vector databases for RAG architectures. Observability & Cost Management: SRE/Platform experience with the practical application of real-time monitoring and cloud ...

technical roadmap for data engineering across Elliptic. Leverage and deploy AI and agentic systems to operate more effectively. Tech environment Scala | Spark | Databricks | AWS | Airflow | Kubernetes | Terraform | Functional Programming You will thrive here if you Enjoy end‐to‐end ownership, from writing the code to architecture design to coaching ...

data platforms, migrating legacy environments into cloud-native solutions using the latest AWS and Snowflake technologies. Key skills: Snowflake, Teradata & Informatica Python & SQL AWS, Airflow & DBT GitHub & CI/CD ETL & Data Warehousing Control-M, Unix/Linux Data Modelling & Performance Optimisation You'll design and optimise scalable data ...

/or Python, and the ability to write complex, efficient SQL, ideally on cloud platforms (e.g. Snowflake). Orchestration & DataOps: Experience with orchestration (e.g. Airflow), git and CI/CD, and an appreciation of FinOps. Cloud: Experience with cloud services, ideally AWS. Agile ways of working: Understanding of Agile ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

Backstage experience - software catalog configuration, scaffolder templates, or plugin development* Familiarity with data platform concepts - medallion architecture, lakehouse patterns, or orchestration tools such as Airflow or dbt* Exposure managing Databricks* Practical experience with AI coding assistants (Claude Code, GitHub Copilot, Cursor, or similar) in a professional engineering context* Experience ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high … commercial experience developing data engineering solutions using Python . Extensive hands-on experience with Databricks , including Workflows, Notebooks and Delta Lake. Strong experience with Apache Spark/PySpark . Proven experience building scalable ETL/ELT pipelines. Strong SQL and data modelling skills. Experience designing modern Lakehouse architectures. Experience ...