1 to 25 of 41 Apache Airflow Jobs in the North of England

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

processing. In this role, you will architect and maintain production‐grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high‐performance solutions. You will collaborate with cross‐functional partners across Finance, Accounting, and Analytics to translate complex business … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine‐tune complex queries ...

deliver secure, reliable, and scalable services that support critical public sector operations. Key Responsibilities Design, develop, and maintain data pipelines and processing workflows using Apache Spark and Python Build, schedule, and monitor workflow orchestration using Apache Airflow Develop and maintain infrastructure-as-code using Terraform across … sprint planning, refinement, and other agile ceremonies Required Skills & Qualifications Experience with AWS services Experience with Terraform for infrastructure as code Experience with Apache Spark for data processing Experience with Apache Airflow for workflow orchestration Experience with Docker and containerisation Experience building and maintaining CI/ ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands-on experience with AWS data services Amazon Redshift or comparable cloud data-warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data-modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands‐on experience with AWS data services Amazon Redshift or comparable cloud data‐warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data‐modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

Amazon Redshift AWS Lambda Amazon Athena Amazon RDS AWS IAM Amazon CloudWatch Experience with services such as Kinesis, EMR, Step Functions or MWAA/Airflow would be advantageous. AWS data engineering roles commonly use services such as S3, Glue, Redshift, Lambda, Kinesis and IAM for data platforms and pipelines. … Automated deployments across environments. Monitoring and logging. Release and configuration management. Production support and incident troubleshooting. Python or another scripting language. AWS certification. Terraform. Apache Airflow/MWAA. Databricks or Spark. Kafka/Kinesis. dbt. Experience working in Agile/Scrum environments. Experience 4+ years of experience ...

required, we are seeking Engineers who have experience with one or more of the following technological areas: Analytics & Platform Frameworks Orchestration like Airflow or AWS Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet ...

test, and deploy ETL pipelines to source, cleanse, and optimise data at scale. Automate and orchestrate data workflows: Develop and maintain orchestration solutions (e.g. Apache Airflow), implementing testing, alerting, monitoring, and data quality checks as first‐class concerns. Ensure data quality and observability: Embed data testing and validation … event streaming platforms including Kafka and its core concepts on topics, messages, and consumers. Experience building and operating automated data workflows using tools like Apache Airflow. Experience implementing data quality and validation checks throughout pipelines to ensure accuracy, completeness, consistency, and reliability. Practical experience implementing monitoring, observability and alerting ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

Following Distributed computing technologies such as Spark and Databricks Cloud platforms including AWS, Azure, or GCP Workflow orchestration tools such as Airflow CI/CD, version control, and infrastructure as code Experience within financial services, consulting, or risk environments What’s on Offer Opportunity to make an impact from ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

required . Proficiency in at least one object-oriented language. Experience with big data technologies such as HDFS, YARN, MapReduce, Hive, Kafka, Spark, Airflow, or Presto. Experience with AWS, GCP, or Looker (advantageous but not essential). Solid background in data modelling, including the design, implementation and optimisation ...

Excellent verbal and written communication skills BS or MS degree in Computer Science or equivalent Experience in distributed computing frameworks like Hive/Hadoop, Apache Spark Experience in developing Finance or HR related applications Experience with following cloud services: AWS Elastic Beanstalk, EC2, S3, CloudFront, RDS, DynamoDB, VPC, Elastic … Cache, Lambda Working experience with Terraform Experience in creating workflows for Apache Airflow What the job involves The Roku Enterprise Engineering team is at the forefront of building a robust, integrated, and scalable platform that powers critical business processes across the organization As a cornerstone of Roku ...

Computer Science or equivalent practical experience Experience developing finance applications is a plus Working experience with Terraform is a plus Experience creating workflows for Apache Airflow is a plus #J-18808-Ljbffr ...

Computer Science or equivalent practical experience Experience developing finance applications is a plus Working experience with Terraform is a plus Experience creating workflows for Apache Airflow is a plus What's Roku's approach to hybrid working? Roku fosters an inclusive and collaborative environment where teams generally work ...

pipelines Strong programming skills in Python and SQL or similar Experience with technologies such as Spark, Databricks, AWS, Azure, GCP and/or Airflow Strong focus on data quality, testing and reliability Solid software engineering fundamentals and high engineering standards Strong communication and collaborative skills Comfortable working with ambiguity ...

practical experience. Preferred Skills Experience with gradient-boosted tree models, neural networks, and optimization solvers in production. Familiarity with data orchestration tools (e.g., Prefect, Airflow, dbt) and modern data lake architectures. Experience working closely with data scientists to operationalize research code. Multi-cloud experience across AWS, GCP, and Azure. ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.We are looking … relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL.* **Orchestration** Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect.* **Data Quality and Lineage** Familiarity with tools and techniques for data testing, documentation and lineage. Examples include Great Expectations or dbt. ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise Observability ...

Skills and knowledge Demonstrable experience designing, deploying, and operating big data platforms at scale. Hands‐on experience with OSS data ecosystems (e.g., Spark, Trino, Airflow, Apache Superset, and similar technologies). Knowledge of cloud platforms and traditional on‐premise hosting; hands‐on experience with Azure, AWS and/ ...

degree in Computer Science, Mathematics, Engineering, Physics or a related discipline from a Russell Group university. Desirable Experience Spark and/or Databricks. Airflow or similar orchestration tools. Distributed computing and large-scale data processing. Defence or National Security experience. Essential Requirements UK national and eligible to obtain ...

configurations for containerised workloads. Implement infrastructure as code using Terraform, covering cloud network, security, compute, and supporting platform resources. Support orchestration tooling such as Airflow as part of the wider platform capability. Manage secrets, configuration, and access controls using Azure Key Vault and Azure Active Directory. Support hybrid connectivity ...

Data Scientists use (we don't expect applicants to have experience with all of these): Python and Databricks for Data Science, Spark, MLFlow and Airflow for ML Workflows, Google Cloud Platform for our analytics infrastructure, dbt and BigQuery for data modelling and warehousing. Discover more about our Analytics Community ...