1 to 25 of 44 Apache Airflow Jobs in the North of England

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Technical Lead – Multi-tenant Data Integration Platform (GCP/Airflow) Lead the engineering delivery and technical direction for a scalable, multi-tenant data integration platform on Google Cloud Platform (GCP). The platform uses Apache Airflow to orchestrate ETL pipelines and enables reliable movement of data from … contribution where needed (critical ETLs, framework components, design spikes, performance fixes). ETL orchestration & data movement Design and develop robust ETL pipelines orchestrated in Apache Airflow , including scheduling strategies, dependency management, retries, backfills, and processing. Establish patterns for onboarding new source systems and delivering data to downstream vendor ...

deliver secure, reliable, and scalable services that support critical public sector operations. Key Responsibilities Design, develop, and maintain data pipelines and processing workflows using Apache Spark and Python Build, schedule, and monitor workflow orchestration using Apache Airflow Develop and maintain infrastructure-as-code using Terraform across … sprint planning, refinement, and other agile ceremonies Required Skills & Qualifications Experience with AWS services Experience with Terraform for infrastructure as code Experience with Apache Spark for data processing Experience with Apache Airflow for workflow orchestration Experience with Docker and containerisation Experience building and maintaining CI/ ...

software development team Hands-on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands-on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … support highly scalable data engineering solutions that power Moodys next-generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moodys technical teams ...

software development team Hands‐on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands‐on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … highly scalable data engineering solutions that power Moody’s next‐generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moody’s technical ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands-on experience with AWS data services Amazon Redshift or comparable cloud data-warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data-modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands‐on experience with AWS data services Amazon Redshift or comparable cloud data‐warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data‐modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

Amazon Redshift AWS Lambda Amazon Athena Amazon RDS AWS IAM Amazon CloudWatch Experience with services such as Kinesis, EMR, Step Functions or MWAA/Airflow would be advantageous. AWS data engineering roles commonly use services such as S3, Glue, Redshift, Lambda, Kinesis and IAM for data platforms and pipelines. … Automated deployments across environments. Monitoring and logging. Release and configuration management. Production support and incident troubleshooting. Python or another scripting language. AWS certification. Terraform. Apache Airflow/MWAA. Databricks or Spark. Kafka/Kinesis. dbt. Experience working in Agile/Scrum environments. Experience 4+ years of experience ...

required, we are seeking Engineers who have experience with one or more of the following technological areas: Analytics & Platform Frameworks Orchestration like Airflow or AWS Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

test, and deploy ETL pipelines to source, cleanse, and optimise data at scale. Automate and orchestrate data workflows: Develop and maintain orchestration solutions (e.g. Apache Airflow), implementing testing, alerting, monitoring, and data quality checks as first‐class concerns. Ensure data quality and observability: Embed data testing and validation … event streaming platforms including Kafka and its core concepts on topics, messages, and consumers. Experience building and operating automated data workflows using tools like Apache Airflow. Experience implementing data quality and validation checks throughout pipelines to ensure accuracy, completeness, consistency, and reliability. Practical experience implementing monitoring, observability and alerting ...

Following Distributed computing technologies such as Spark and Databricks Cloud platforms including AWS, Azure, or GCP Workflow orchestration tools such as Airflow CI/CD, version control, and infrastructure as code Experience within financial services, consulting, or risk environments What’s on Offer Opportunity to make an impact from ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

required . Proficiency in at least one object-oriented language. Experience with big data technologies such as HDFS, YARN, MapReduce, Hive, Kafka, Spark, Airflow, or Presto. Experience with AWS, GCP, or Looker (advantageous but not essential). Solid background in data modelling, including the design, implementation and optimisation ...

design, build, and support scalable data pipelines powering Moody’s next-generation digital content platform. You will develop data workflows using Databricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS to turn diverse data sources into reliable analytics. You will collaborate with cross-functional teams and mentor ...

Computer Science or equivalent practical experience Experience developing finance applications is a plus Working experience with Terraform is a plus Experience creating workflows for Apache Airflow is a plus What's Roku's approach to hybrid working? Roku fosters an inclusive and collaborative environment where teams generally work ...

pipelines Strong programming skills in Python and SQL or similar Experience with technologies such as Spark, Databricks, AWS, Azure, GCP and/or Airflow Strong focus on data quality, testing and reliability Solid software engineering fundamentals and high engineering standards Strong communication and collaborative skills Comfortable working with ambiguity ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise Observability ...

workstreams or managing delivery teams. Working within highly regulated or public sector environments. Experience with technologies such as Databricks, Snowflake, Fabric, Spark, Data Factory, Airflow or dbt would be highly advantageous. Please note that SC Clearance is required for this position Apply now or send your CV directly ...

workstreams or managing delivery teams. Working within highly regulated or public sector environments. Experience with technologies such as Databricks, Snowflake, Fabric, Spark, Data Factory, Airflow or dbt would be highly advantageous. Please note that SC Clearance is required for this position Apply now or send your CV directly ...

workstreams or managing delivery teams. Working within highly regulated or public sector environments. Experience with technologies such as Databricks, Snowflake, Fabric, Spark, Data Factory, Airflow or dbt would be highly advantageous. Please note that SC Clearance is required for this position Apply now or send your CV directly ...

practical experience. Preferred Skills Experience with gradient‐boosted tree models, neural networks, and optimization solvers in production. Familiarity with data orchestration tools (e.g., Prefect, Airflow, dbt) and modern data lake architectures. Experience working closely with data scientists to operationalize research code. Multi‐cloud experience across AWS, GCP, and Azure. ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.We are looking … relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL.* **Orchestration** Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect.* **Data Quality and Lineage** Familiarity with tools and techniques for data testing, documentation and lineage. Examples include Great Expectations or dbt. ...

data engineering. You'll likely have experience with: Python SQL AWS, Azure or GCP Building production data pipelines Data modelling and data platforms Git Airflow or similar orchestration technologies Testing, debugging and improving data reliability We're particularly interested in engineers who have worked in consultancy, professional services ...