1 to 25 of 44 Apache Airflow Jobs in the North of England

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

processing. In this role, you will architect and maintain production‐grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high‐performance solutions. You will collaborate with cross‐functional partners across Finance, Accounting, and Analytics to translate complex business … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine‐tune complex queries ...

deliver secure, reliable, and scalable services that support critical public sector operations. Key Responsibilities Design, develop, and maintain data pipelines and processing workflows using Apache Spark and Python Build, schedule, and monitor workflow orchestration using Apache Airflow Develop and maintain infrastructure-as-code using Terraform across … sprint planning, refinement, and other agile ceremonies Required Skills & Qualifications Experience with AWS services Experience with Terraform for infrastructure as code Experience with Apache Spark for data processing Experience with Apache Airflow for workflow orchestration Experience with Docker and containerisation Experience building and maintaining CI/ ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands‐on experience with AWS data services Amazon Redshift or comparable cloud data‐warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data‐modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

Amazon Redshift AWS Lambda Amazon Athena Amazon RDS AWS IAM Amazon CloudWatch Experience with services such as Kinesis, EMR, Step Functions or MWAA/Airflow would be advantageous. AWS data engineering roles commonly use services such as S3, Glue, Redshift, Lambda, Kinesis and IAM for data platforms and pipelines. … Automated deployments across environments. Monitoring and logging. Release and configuration management. Production support and incident troubleshooting. Python or another scripting language. AWS certification. Terraform. Apache Airflow/MWAA. Databricks or Spark. Kafka/Kinesis. dbt. Experience working in Agile/Scrum environments. Experience 4+ years of experience ...

pipelines and warehouses. Strong SQL skills and proficiency in Python or another relevant programming language. Experience with modern data engineering tools, such as dbt, Airflow or Spark. Experience with cloud data warehouses, such as Snowflake, BigQuery or Amazon Redshift. A solid understanding of data modelling, ETL/ ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

required, we are seeking Engineers who have experience with one or more of the following technological areas: Analytics & Platform Frameworks Orchestration like Airflow or AWS Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

relational database design Engineering-SQL, Python, ETL/ELT, APIs, data modelling, data warehousing, testing and observability Orchestration-Azure Data Factory, Prefect, Airflow or comparable workflow platforms Business intelligence-Power BI, SSRS, QlikView, semantic modelling and governed self‐service reporting Development-Git/GitHub, CI/CD, VB/ ...

version control R experience is mandatory for this role. Desirable experience MySQL Python Docker and containerisation CI/CD pipelines Terraform or CloudFormation Airflow, AWS Step Functions or other workflow orchestration tools Building APIs or packaging models in R, such as plumber or R packages SaaS or software product ...

Following Distributed computing technologies such as Spark and Databricks Cloud platforms including AWS, Azure, or GCP Workflow orchestration tools such as Airflow CI/CD, version control, and infrastructure as code Experience within financial services, consulting, or risk environments What’s on Offer Opportunity to make an impact from ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

Computer Science or equivalent practical experience Experience developing finance applications is a plus Working experience with Terraform is a plus Experience creating workflows for Apache Airflow is a plus #J-18808-Ljbffr ...

Computer Science or equivalent practical experience Experience developing finance applications is a plus Working experience with Terraform is a plus Experience creating workflows for Apache Airflow is a plus What's Roku's approach to hybrid working? Roku fosters an inclusive and collaborative environment where teams generally work ...

pipelines Strong programming skills in Python and SQL or similar Experience with technologies such as Spark, Databricks, AWS, Azure, GCP and/or Airflow Strong focus on data quality, testing and reliability Solid software engineering fundamentals and high engineering standards Strong communication and collaborative skills Comfortable working with ambiguity ...

Excellent verbal and written communication skills BS or MS degree in Computer Science or equivalent Experience in distributed computing frameworks like Hive/Hadoop, Apache Spark Experience in developing Finance or HR related applications Experience with following cloud services: AWS Elastic Beanstalk, EC2, S3, CloudFront, RDS, DynamoDB, VPC, Elastic … Cache, Lambda Working experience with Terraform Experience in creating workflows for Apache Airflow What the job involves The Roku Enterprise Engineering team is at the forefront of building a robust, integrated, and scalable platform that powers critical business processes across the organization As a cornerstone of Roku ...

problems and design solutions that meet their objectives. Additionally, you will be responsible for designing, developing, and operating a data service platform using Python, Airflow, and SQL to build the various ETL, analytics, and data quality components. You’ll automate deployments using AWS CodeDeploy, AWS CodePipeline, AWS Cloud Development ...

practical experience. Preferred Skills Experience with gradient-boosted tree models, neural networks, and optimization solvers in production. Familiarity with data orchestration tools (e.g., Prefect, Airflow, dbt) and modern data lake architectures. Experience working closely with data scientists to operationalize research code. Multi-cloud experience across AWS, GCP, and Azure. ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.We are looking … relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL.* **Orchestration** Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect.* **Data Quality and Lineage** Familiarity with tools and techniques for data testing, documentation and lineage. Examples include Great Expectations or dbt. ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise Observability ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise • Observability ...

necessary infrastructure, resources, and interfaces to enable data loading and LLM workflows. Use Python/Java and large-scale data workflow orchestration platforms e.g. Airflow to construct software artifacts for ETL, interfacing with diverse data formats and storage technologies, and incorporate them into robust data workflows and dynamic systems ...

degree in Computer Science, Mathematics, Engineering, Physics or a related discipline from a Russell Group university. Desirable Experience Spark and/or Databricks. Airflow or similar orchestration tools. Distributed computing and large-scale data processing. Defence or National Security experience. Essential Requirements UK national and eligible to obtain ...