1 to 25 of 79 Apache Airflow Jobs in the UK

What you'll be doing: * Building and maintaining scalable data pipelines and workflows * Developing Python based solutions for data processing and integration * Working with Apache Airflow to orchestrate data workflows * Supporting containerised environments using Docker * Collaborating with engineers, analysts, and client teams on technical delivery * Contributing to continuous … improvement and platform reliability What you'll bring: * Strong hands on experience with Python 3 * Experience working with Apache Airflow * Experience working within Dockerised environments * Good understanding of data engineering principles and ETL processes * Strong problem solving and communication skills * Financial markets or trading domain exposure would ...

What you'll be doing: * Building and maintaining scalable data pipelines and workflows * Developing Python based solutions for data processing and integration * Working with Apache Airflow to orchestrate data workflows * Supporting containerised environments using Docker * Collaborating with engineers, analysts, and client teams on technical delivery * Contributing to continuous … improvement and platform reliability What you'll bring: * Strong hands on experience with Python 3 * Experience working with Apache Airflow * Experience working within Dockerised environments * Good understanding of data engineering principles and ETL processes * Strong problem solving and communication skills * Financial markets or trading domain exposure would ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

data validation and application testing processes across distributed systems Investigate production issues using data analysis to identify root causes Work with orchestration tools (e.g., Apache Airflow) to monitor workflows and troubleshoot failures Validate data using tools such as Jupyter Notebooks and Amazon Athena Analyse data across multiple formats … additional programming language (e.g., Java) Experience with scripting languages such as Bash Strong understanding of automated testing frameworks and methodologies Hands-on experience with Apache Airflow and log analysis Familiarity with data processing tools such as Apache Spark Experience working with cloud platforms and services, including: Monitoring ...

automate workflows, and deliver secure, high-performance data platforms using AWS-native technologies. Key Responsibilities Develop and maintain data pipelines using Python , SQL , and Apache Airflow . Build and manage containerised data applications using Docker . Design and optimise solutions using AWS services including Redshift, OpenSearch, Lambda, Glue … data governance, security, and best practice standards. Essential Skills Active Security Clearance Strong experience with Python and SQL . Hands-on experience with Apache Airflow and Docker . Proven experience with AWS data services, including Redshift, OpenSearch, Lambda, Glue, Step Functions, and Batch . Experience with Git , GitLab ...

environment. You'll have the opportunity to contribute to AI initiatives, support evolving data platforms, and work with emerging technologies including Microsoft Fabric, GitHub, Apache Airflow, and intelligent automation tooling. The team offers a supportive culture, strong learning opportunities, and clear pathways for career progression and technical growth. … analytics teams with high-quality data engineering practices. Contribute to strategic platform modernisation initiatives including AI projects, Microsoft Fabric exploration, GitHub adoption, and Apache Airflow implementation. Work within a collaborative Azure-based data engineering environment, bringing strong SQL and general data engineering expertise; SAS experience is beneficial ...

Labelled property graphs Hypergraph structures Entity resolution Graph schema design Data Engineering & Integration Pipeline Development Data Integration Languages & Tools ETL/ELT pipeline design Apache Kafka Apache Spark AWS Glue dbt Apache Airflow Entity matching and resolution Data lineage tracking Schema mapping Semantic data integration Metadata ...

producing a clear blueprint for future delivery. Key responsibilities include: * Mapping and documenting the current data architecture and reporting ecosystem * Analysing existing Python and Apache Airflow data pipelines * Tracing data flows from multiple partner data sources through Google Cloud environments into reporting platforms * Reviewing and documenting transformation logic … platforms * Excellent documentation and communication skills * Experience with Tableau and Power BI environments * Knowledge of Google Cloud Platform (GCP) * Familiarity with PostgreSQL, Python and Apache Airflow * Experience within digital media, publishing, advertising technology or affiliate commerce environments * Understanding of programmatic advertising and revenue reporting models * Previous experience supporting ...

producing a clear blueprint for future delivery. Key responsibilities include: * Mapping and documenting the current data architecture and reporting ecosystem * Analysing existing Python and Apache Airflow data pipelines * Tracing data flows from multiple partner data sources through Google Cloud environments into reporting platforms * Reviewing and documenting transformation logic … platforms * Excellent documentation and communication skills * Experience with Tableau and Power BI environments * Knowledge of Google Cloud Platform (GCP) * Familiarity with PostgreSQL, Python and Apache Airflow * Experience within digital media, publishing, advertising technology or affiliate commerce environments * Understanding of programmatic advertising and revenue reporting models * Previous experience supporting ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Notebooks for data exploration and development Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

career in data engineering, it can feel like youre staring at a never-ending list of tools and technologies SQL, Python, Spark, Kafka, Airflow, dbt, Snowflake, Redshift, Terraform, Kubernetes, and the list goes on. Scroll job boards and LinkedIn, and its easy to conclude that unless you have experience ...

career in data engineering, it can feel like youre staring at a never-ending list of tools and technologies SQL, Python, Spark, Kafka, Airflow, dbt, Snowflake, Redshift, Terraform, Kubernetes, and the list goes on. Scroll job boards and LinkedIn, and its easy to conclude that unless you have experience ...

Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills with ...

Glue, ECR, EC2, EMR, DynamoDB, Lake Formation Experience with Infrastructure as Code (Terraform) Proficiency in Python, SQL, and familiarity with PySpark Experience with Apache Spark and data processing frameworks Hands-on experience with Apache Airflow for workflow orchestration Experience using Jupyter Notebooks and/or Amazon Athena ...

technical stakeholders Strong communication, problem-solving and analytical skills Desirable Experience Experience with Azure, AWS or GCP Exposure to orchestration tools such as Airflow, Dagster or Fabric Pipelines Experience with Git, CI/CD pipelines and DataOps practices Knowledge of Microsoft Purview, Unity Catalog or similar governance tools Experience ...

model on hybrid/on-prem environments. Hard Skills Experience in data engineering, ideally aerospace/defence or industrial sectors. ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka). Strong Python and SQL; C++, C#, or MATLAB a plus. Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg ...

Spark, Databricks, Hadoop or cloud-native alternatives). Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow). Strong experience with data engineering concepts (ETL/ELT, data integration, warehousing, vector databases/search) A little bit more about us... ...

/EKS, Step Functions). Experience with MLOps tooling such as: Experiment tracking and model registries (MLflow, Weights & Biases, SageMaker Model Registry) Workflow orchestration (Airflow, Prefect, Step Functions) Model serving frameworks (SageMaker, TorchServe, TensorFlow Serving, or similar) Feature stores (Feast, Tecton, or custom-built) Deep understanding of monitoring ...

analytics and reporting, and contributing to the adoption of AI-powered workflows and automation. Key Responsibilities & Skills: Build and maintain scalable data pipelines using Apache Airflow or similar orchestration tools. Develop integrations with internal and third-party APIs, ensuring reliable and efficient data flows. Design and optimise ...

ensuring data integrity across every layer Lead the transformation of legacy reporting (Python, SQL, Excel) into a secure,scalableand fully observable platform powered by Apache Airflow (WMAA). Define and publish data quality metrics that drive engineering decisions anddemonstrateplatform health to stakeholders Champion AI and modern data tooling ...

with a strong focus on ETL/ELT. • Advanced SQL and Python development skills. • Hands-on experience with DBT, GIT, Terraform, Docker, IAM, and Airflow (Composer). • Proven experience working on cloud platforms – ideally GCP (BigQuery), but Azure or AWS also considered. • Strong understanding of Data Mesh, Test Driven ...

including optimisation of complex queries (E). Experience building data pipelines and ETL/ELT workflows using tools such as: Azure Data Factory, Databricks, Airflow, Luigi, or similar (E) Strong understanding of data modelling (E) Programming skills in Python and/or Scala for data processing (D). Experience ...

/T-SQL with proven experience refactoring and optimising complex logic Hands-on experience with cloud data warehouses (e.g. BigQuery, Snowflake), ingestion tools (Airflow, Airbyte), and analytics tooling such as dbt and Power BI Strong working knowledge of Git/version control, CI/CD pipelines, and Python ...

data validation frameworks Experience of testing data platforms such as Snowflake, BigQuery, Redshift, Databricks, Azure Synapse Experience of testing pipelines in things like Airflow, Azure Data Factory, AWS Glue, Kafka Data Concepts ETL/ELT, data modelling, data warehousing, data governance The role is 100% remote, however you must ...

preparation specifically geared toward AI model integration Technical Fluency: A strong conceptual understanding of modern data ecosystems, data warehousing, pipeline orchestration tools (e.g., dbt, Airflow), and data governance standards Cross-Functional Communication: Outstanding stakeholder management skills, with a demonstrated ability to foster collaboration between technical developers and non-technical ...