1 to 25 of 51 Apache Airflow Jobs in London

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … through to validated delivery Architect and build the Claude Code agentic workflow: Parse existing data movement pipelines and logic Generate equivalent PySpark transformations Produce Airflow DAGs Operate, iterate, and scale across the full data pipeline estate Ensure rigorous creation and management of the PySpark data processing layer within ...

Labelled property graphs Hypergraph structures Entity resolution Graph schema design Data Engineering & Integration Pipeline Development Data Integration Languages & Tools ETL/ELT pipeline design Apache Kafka Apache Spark AWS Glue dbt Apache Airflow Entity matching and resolution Data lineage tracking Schema mapping Semantic data integration Metadata ...

producing a clear blueprint for future delivery. Key responsibilities include: * Mapping and documenting the current data architecture and reporting ecosystem * Analysing existing Python and Apache Airflow data pipelines * Tracing data flows from multiple partner data sources through Google Cloud environments into reporting platforms * Reviewing and documenting transformation logic … platforms * Excellent documentation and communication skills * Experience with Tableau and Power BI environments * Knowledge of Google Cloud Platform (GCP) * Familiarity with PostgreSQL, Python and Apache Airflow * Experience within digital media, publishing, advertising technology or affiliate commerce environments * Understanding of programmatic advertising and revenue reporting models * Previous experience supporting ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Notebooks for data exploration and development Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ETL/ELT development and orchestration tools (e.g., Apache Airflow, dbt, Prefect). Knowledge of data modelling, data warehousing, and lakehouse architectures. Familiarity with DevOps practices, CI/CD pipelines, and infrastructure ...

public sector clients. Key Skills 3+ years' Data Engineering experience Strong Python and SQL skills ETL/ELT pipeline development Experience with Spark, Kafka, Airflow or similar technologies Cloud experience (AWS, Azure or GCP) CI/CD and DevOps tooling Data modelling and database design Strong stakeholder and communication ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud Storage … . Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Master's degree in Computer Science, Engineering, Information Technology, or a related field. ...

solutions • Knowledge of DevOps practices including CI/CD and Infrastructure as Code (e.g. Terraform) The other stuff we are looking for • Experience with Apache Airflow or Cloud Composer for orchestration • Exposure to Agile ways of working • Knowledge of Kubernetes or Dataflow • Familiarity with software engineering principles such ...

working in secure, regulated, or complex enterprise environments Exposure to large-scale data platforms or transformation programmes Knowledge of tools such as Spark, Databricks, Airflow, or similar Experience in consulting or client-facing delivery environments Security Requirements Must be a sole UK citizen Must be eligible for Security Clearance ...

data platforms (e.g. Snowflake) • Data Modelling Skills: Proven ability to design scalable, well-structured and reusable data models • Tooling Experience: Experience with dbt, Python, Airflow or similar modern data stack tools • DataOps Practices: Familiarity with git, CI/CD and testing frameworks for data pipelines • Data Quality Focus: Strong ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

with experience building and maintaining production systems Experience with Kubernetes - deploying, managing, and troubleshooting containerised workloads Familiarity with ML platform tooling such as MLflow, Airflow, or similar orchestration and experiment tracking frameworks Experience with cloud infrastructure (AWS or GCP) including compute, storage, and networking Understanding of distributed systems principles ...

Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/GCP) Experience … with modern data warehousing solutions (e.g., Snowflake, BigQuery, Redshift) Familiarity with orchestration tools (e.g., Airflow, Databricks Workflows) Strong understanding of data modelling and lakehouse architecture Databricks certifications (e.g., Databricks Certified Data Engineer) Experience with streaming data pipelines (Kafka, Structured Streaming) Experience with CI/CD and DevOps practices Knowledge ...

real-time processing. Build and maintain data ingestion frameworks from multiple data sources. Develop scalable data models and data warehouse solutions. Implement workflow orchestration (Airflow/Cloud Composer) including retries, dependencies, and monitoring. Ensure data quality, validation, and governance mechanisms. Design and manage streaming pipelines (Kafka/ ...

Python for data engineering and strong SQL skills for data modelling and analysis Experience building and orchestrating data pipelines using tools such as Airflow or dbt within cloud-based environments Expertise in developing clear, user-friendly dashboards using tools such as Tableau or Looker Studio Deep understanding of music ...

environments Solid experience with SQL and data modelling concepts Strong analytical and problem-solving skills Nice to Have Skills Experience with data pipeline tools (Airflow, Kafka, Spark) Knowledge of cloud platforms (AWS, Azure or GCP) Familiarity with relational and NoSQL databases Additional programming experience (Python, Scala) Essential Requirements Active ...

times series data, they use TimescaleDB but other experience with similar platforms is fine You have experience with Orchestration platforms such as Dagster or Airflow You're collaborative with great communication skills What's in it for you: Salary to £120k + bonus X3 weeks per year work anywhere ...

Nice to Have: * Microsoft Azure Solutions Architect Expert* AWS Solutions Architect Professional* Databricks Certifications* CDMP Certification* Terraform, Bicep or Infrastructure-as-Code experience* dbt, Airflow or modern orchestration frameworks* Experience supporting AI, GenAI or RAG-based architectures* Experience within Financial Services, Healthcare, Life Sciences, Energy, Retail or Manufacturing Robert ...

Looker Strong communication skills and confidence working with non-technical stakeholders Familiarity with AI tools to accelerate your own workflow Preferred Skills Dagster or Airflow experience is a nice to have Any exposure to ML workflows or real-time pipelines Pay range and compensation package ...

Snowflake Toolkit: Hands-on with Snowpipe, Streams, Tasks, Dynamic Tables, and Stored Procedures. Nice to Have: Exposure to DBT, Snowpark, Snowflake AI/ML, Airflow, and Snowflake administration (RBAC, warehouse sizing). Mindset: Strong problem-solving abilities with an ownership mindset for end-to-end delivery. This ...

production-grade data pipelines Solid understanding of integration patterns including REST APIs, webhooks, OAuth and event-driven architectures Experience with orchestration tools such as Airflow, Prefect or Dagster Familiarity with cloud environments (AWS, GCP or Azure) Experience with containerisation tools such as Docker and Kubernetes Proven experience integrating multiple ...

necessary infrastructure, resources, and interfaces to enable data loading and LLM workflows. Use Python/Java and large-scale data workflow orchestration platforms e.g. Airflow to construct software artifacts for ETL, interfacing with diverse data formats and storage technologies, and incorporate them into robust data workflows and dynamic systems ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

Experience working within cloud environments (AWS or Azure) Strong SQL skills and experience working with large-scale datasets Familiarity with orchestration tools such as Airflow or similar Good understanding of data modelling and distributed data processing concepts Comfortable working across multiple client environments and stakeholders 🎁 What’s on Offer ...

preferred) or similar platforms: Snowflake, BigQuery, Redshift, Synapse, or equivalent Strong proficiency with: SQL (advanced level) Spark/PySpark Data pipeline orchestration tools (e.g., Airflow, ADF, etc.) Data Modelling Extensive experience designing: Fact and dimension tables Star and snowflake schemas Aggregation layers and semantic models Data Migration & Mapping Demonstrated ...