1 to 25 of 143 Permanent Apache Airflow Jobs in the UK excluding London

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Technical Lead – Multi-tenant Data Integration Platform (GCP/Airflow) Lead the engineering delivery and technical direction for a scalable, multi-tenant data integration platform on Google Cloud Platform (GCP). The platform uses Apache Airflow to orchestrate ETL pipelines and enables reliable movement of data from … contribution where needed (critical ETLs, framework components, design spikes, performance fixes). ETL orchestration & data movement Design and develop robust ETL pipelines orchestrated in Apache Airflow , including scheduling strategies, dependency management, retries, backfills, and processing. Establish patterns for onboarding new source systems and delivering data to downstream vendor ...

Title Technical Lead – Multi-tenant Data Integration Platform (GCP/Airflow) Role purpose Lead the engineering delivery and technical direction for a scalable, multi-tenant data integration platform on Google Cloud Platform (GCP). The platform uses Apache Airflow to orchestrate ETL pipelines and enables reliable movement … contribution where needed (critical ETLs, framework components, design spikes, performance fixes). ETL orchestration & data movement Design and develop robust ETL pipelines orchestrated in Apache Airflow , including scheduling strategies, dependency management, retries, backfills, and processing. Establish patterns for onboarding new source systems and delivering data to downstream vendor ...

deliver secure, reliable, and scalable services that support critical public sector operations. Key Responsibilities Design, develop, and maintain data pipelines and processing workflows using Apache Spark and Python Build, schedule, and monitor workflow orchestration using Apache Airflow Develop and maintain infrastructure-as-code using Terraform across … sprint planning, refinement, and other agile ceremonies Required Skills & Qualifications Experience with AWS services Experience with Terraform for infrastructure as code Experience with Apache Spark for data processing Experience with Apache Airflow for workflow orchestration Experience with Docker and containerisation Experience building and maintaining CI/ ...

software development team Hands-on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands-on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … support highly scalable data engineering solutions that power Moodys next-generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moodys technical teams ...

software development team Hands‐on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands‐on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … highly scalable data engineering solutions that power Moody’s next‐generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moody’s technical ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands-on experience with AWS data services Amazon Redshift or comparable cloud data-warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data-modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

designing and building robust, scalable, distributed data systems and pipelines, using open source and public cloud technologies. Strong experience with data orchestration tools: e.g. Apache Airflow, Dagster. Experience with big data storage and processing technologies: e.g. DBT, Spark, SQL, Athena/Trino, Redshift, Snowflake, RDBMSs (PostgreSQL/MySQL … . Knowledge of event‐driven architectures and streaming technologies: e.g. Apache Kafka, Kafka Streams, Apache Flink. Experience with public cloud environments: e.g. AWS, GCP, Azure, Terraform. Strong knowledge of software engineering practices: e.g. testing, CI/CD (Jenkins, Github Actions), agile development, git/version control, containers etc. ...

Senior Data Engineer - Python, SQL, Airflow, Azure, Pyspark, Kubernetes Salary: Up to £72,000 per annum Benefits: Excellent benefits package including annual bonus, free onsite gym, and company pension scheme. About the Role We are seeking an experienced Senior Data Engineer to join an innovative SaaS business that … Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

fields of data analytics and software Proficiency in Python programming language and common data science packages. Experience with data engineering tools (e.g. Apache Spark, AWS Glue, Kinesis, Snowflake, Apache Airflow) Experience in building data pipelines and data streaming applications. Experience with tools to support the software ...

stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Familiarity with additional technologies: GraphQL, React, HTML5, JavaScript, CSS, Postgres, Gradle, BERT. Experience working with and designing infrastructures for large ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

modern data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python. Experience with data engineering and pipeline technologies such as Apache Airflow and Spark. Experience working with cloud platforms, particularly AWS and Azure. Knowledge of big data technologies and distributed data processing. Understanding ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

ideally with a background in software engineering, product management or data analytics. Strong proficiency in Python or Java, SQL, and data pipeline frameworks (e.g., Airflow, dbt, Spark), with solid experience with the AWS ecosystem. Excellent problem-solving skills and ability to work in a fast-paced environment. Bachelor ...

ideally with a background in software engineering, product management or data analytics. Strong proficiency in Python or Java, SQL, and data pipeline frameworks (e.g., Airflow, dbt, Spark), with solid experience with the AWS ecosystem. Excellent problem-solving skills and ability to work in a fast-paced environment. Bachelor ...