1 to 25 of 99 Permanent Apache Airflow Jobs in the UK

looking for a Test Engineer to join a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB. You will own testing across automated pipelines, data workflows, and cloud infrastructure - identifying risks, championing … Development Identify, evaluate, and implement new test frameworks Enhance existing frameworks to improve testing confidence and coverage Pipeline & Data Testing Validate data pipelines using Apache Airflow, AWS Glue, Athena, and EMR Ensure data integrity, transformation accuracy, and performance under load Analyse data in multiple formats to validate ...

create the queries and processes necessary to move everything to the new. As migration work wraps up, the role will transition to leveraging Airflow and other tooling to automate operational and financial processes. This is a great role for exceptionally outgoing individuals as all work will require extensive collaboration … design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process ...

create the queries and processes necessary to move everything to the new. As migration work wraps up, the role will transition to leveraging Airflow and other tooling to automate operational and financial processes. This is a great role for exceptionally outgoing individuals as all work will require extensive collaboration … design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality of coding ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality of coding ...

choose the appropriate tools. Below are key tools/technologies that we currently use in our team: · Java 17/21, Spring boot · Python (Apache Airflow/Apache Spark) · Typescrpt (Lambda, AWS CDK) · AWS: S3, MSK (Kafka), Glue, Athena, Flink · CI/CD: Jenkins, Github Actions ...

data warehousing, dimensional modelling and medallion architecture Hands-on experience with cloud platforms: AWS, Azure or GCP Experience with orchestration tools such as Airflow, Prefect or Dagster Strong SQL skills and experience with modern warehouse/lakehouse architectures Familiarity with Git, CI/CD and infrastructure-as-code practices ...

with experience in CI/CD pipelines, monitoring, and observability tools (Grafana or equivalent). Exposure to analytics, reporting, and BI tools such as Apache Superset, Lightdash or OpenSearch Willingness to work across the stack by contributing to API development and, at times, UI components (Vue.js, Zoho, or similar ...

Proven experience with modern data platforms such as Snowflake (AI Data Cloud) and cloud‐native services. Good understanding of open‐source table formats, specifically Apache Iceberg , to enable interoperability, schema evolution, and high‐performance analytics across multiple engines. Programming: Expert‐level proficiency in Java , Python and SQL . … Data & Orchestration: Hands‐on experience with Spark, Kafka, and orchestration tools like Apache Airflow, Dagster, or dbt. Data Modeling: Deep understanding of data warehousing and modern data lakehouse architecture. Leadership & Soft Skills Mentorship: Proven track record of upskilling junior and senior engineers. Communication: Ability to explain complex technical ...

clear, structured data engineering framework Build, enhance, and maintain data pipelines within Snowflake Develop transformations and data models using DBT Orchestrate workflows using Apache Airflow Write clean, modular Python code for data engineering solutions Improve pipeline reliability, performance, and observability Introduce and embed CI/CD, testing … Lead level Extensive experience with Snowflake as a core data platform Advanced SQL with strong business logic and modelling capability Experience with DBT, Airflow, and Python Experience building modular, scalable solutions within Snowflake CI/CD pipeline experience, ideally using Azure DevOps Exposure to Docker and modern software engineering ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

Desirable Substantial experience in senior information role in an NHS or other healthcare information environment Experience with modern data engineering tools and frameworks (e.g. Airflow, Spark, Snowflake etc.) Proven track record of staff management, change management, and performance management Skills and Abilities Essential Knowledge of information analysis ...

stream data processing. Cloud Pub/Sub: For real-time messaging and event ingestion. Cloud Storage: For scalable and durable object storage. Cloud Composer (Apache Airflow): For workflow orchestration and pipeline management. Cloud Dataproc: For managed Hadoop and Spark clusters (good to have). Cloud SQL/Cloud ...

/GCP beneficial) Strong data modelling knowledge Understanding of data governance and security best practices Ability to translate business needs into scalable solutions Desirable Airflow, dbt, Kafka, CI/CD, BI tools, governance platforms, financial services experience. Interested in building modern, enterprise-scale data platforms in a highly regulated ...

data ingestion pipelines; advanced SQL and data modelling skills; experience with cloud platforms (AWS, GCP, or Azure); familiarity with pipeline orchestration tools such as Airflow or Prefect. Nice to Have: Experience working with financial, operational, or transactional data; exposure to real-time or event-driven systems; experience with data ...

driving innovation in data engineering. Desirable Criteria Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. Familiarity with Apache Airflow, data modelling, and metadata management Experience influencing enterprise data strategy and contributing to architectural governance. To apply for this position please submit ...

pipelines Familiarity with CI/CD, Git, Terraform, and modern engineering practices Experience working in Agile/Scrum environments Exposure to tools such as Airflow/Composer and Azure DevOps Understanding of AI/ML tooling such as Vertex AI Desirable Experience with BI/visualisation tools (e.g. Looker ...

delivering customer-facing Enterprise SaaS products at scale. Strong hands-on or architectural experience with modern data platform tooling, including but not limited to Apache Airflow, dbt, Kafka, AWS Glue, AWS Redshift, AWS Sagemaker, AWS Dynamo, AWS Quick Sight. Success leading a data engineering team through complex challenges ...

building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps , CI/ ...

Azure (experience with AWS or GCP a plus). Strong technical proficiency with the modern data engineering stack, including tools such as dbt, Coalesce, Airflow, Fivetran, and Python. Familiarity with DataOps practices, including CI/CD and infrastructure-as-code principles. Demonstrated success managing distributed teams across multiple geographies. ...

Azure (experience with AWS or GCP a plus). Strong technical proficiency with the modern data engineering stack, including tools such as dbt, Coalesce, Airflow, Fivetran, and Python. Familiarity with DataOps practices, including CI/CD and infrastructure-as-code principles. Demonstrated success managing distributed teams across multiple geographies. ...

architectural strategies, blueprints for hybrid and cloud-native solutions ELT/ETL Frameworks & Pipelines Essential Develop robust ELT/ETL pipelines using tools like Apache Airflow, DBT, AWS Glue, Azure Data Factory, or Kafka Connect. Desirable Optimize data transformations for performance, reusability, and modular design (e.g., using ...

Glue or EKS Hands on experience with modern analytics stacks such as Snowflake, dbt and Iceberg Familiarity with orchestration tools such as Dagster or Airflow Strong understanding of data modelling concepts and analytics best practice A software engineering mindset, including Git, testing and CI/CD workflows What They ...

Glue or EKS Hands on experience with modern analytics stacks such as Snowflake, dbt and Iceberg Familiarity with orchestration tools such as Dagster or Airflow Strong understanding of data modelling concepts and analytics best practice A software engineering mindset, including Git, testing and CI/CD workflows What They ...

Glue or EKS Hands on experience with modern analytics stacks such as Snowflake, dbt and Iceberg Familiarity with orchestration tools such as Dagster or Airflow Strong understanding of data modelling concepts and analytics best practice A software engineering mindset, including Git, testing and CI/CD workflows What They ...