1 to 25 of 362 Permanent Apache Airflow Jobs in England

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Consultant to support the implementation and deployment of Astronomer (Astro) within a leading asset management organisation. Job Responsibilities/Objectives Astronomer is built around Apache Airflow and is used to develop, schedule, automate and manage complex data pipelines. The organisation is currently implementing the platform and requires … ownership of the implementation and providing training and support to upskill the wider team. Experience with Astronomer/Astro is essential, alongside knowledge of Apache Airflow, dbt and Snowflake. Key Responsibilities Lead the implementation, build and deployment of Astronomer (Astro). Configure and manage Apache Airflow ...

seeking a Technical Delivery Lead to drive the engineering and technical strategy for a scalable, multi-tenant data integration platform built on Apache Airflow and Google Cloud. In this role, you will lead the delivery of production-grade data pipelines and cloud solutions, working closely with cross-functional … guiding teams in a collaborative environment. Roles and Responsibilities Lead the engineering and technical strategy for a scalable multi‐tenant data integration platform leveraging Apache Airflow and GCP technologies. Build and maintain production‐grade ETL/ELT pipelines orchestrated with Apache Airflow. Utilize container platforms such ...

processing. In this role, you will architect and maintain production‐grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high‐performance solutions. You will collaborate with cross‐functional partners across Finance, Accounting, and Analytics to translate complex business … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine‐tune complex queries ...

deliver secure, reliable, and scalable services that support critical public sector operations. Key Responsibilities Design, develop, and maintain data pipelines and processing workflows using Apache Spark and Python Build, schedule, and monitor workflow orchestration using Apache Airflow Develop and maintain infrastructure-as-code using Terraform across … sprint planning, refinement, and other agile ceremonies Required Skills & Qualifications Experience with AWS services Experience with Terraform for infrastructure as code Experience with Apache Spark for data processing Experience with Apache Airflow for workflow orchestration Experience with Docker and containerisation Experience building and maintaining CI/ ...

bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than … this role: Astronomer is on a mission to make DataOps a first-class discipline in every modern data organization. As the driving force behind Apache Airflow, we're powering mission-critical pipelines at scale for companies like Condé Nast, Rappi, Sonos, and more. As a Sales Engineer ...

Linux operating systems, including command-line administration and troubleshooting. Solid experience with SQL or data pipeline technologies, Experience with data processing frameworks such as Apache Airflow, Apache Flink or Google Dataflow, with a practical understanding of ETL pipeline design and the ability to diagnose issues across data …/IP networking principles, including the ability to diagnose connectivity issues and interpret network traffic. Familiarity with event streaming platforms such as Apache Kafka, including an understanding of producer/consumer patterns and how message queues are used to move data reliably between systems at scale. Familiarity with infrastructure ...

London --------------------------------------- JOB DETAILS Role Title: Data Engineer (Python Enterprise Developer) Must have skills: -Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow -Preferred Domain: Energy trading Minimum number of relevant years of experience: 8+ Detailed Job Description: JD: • 8+ years of experience in python … communication skills with an ability to lead a team and keep them motivated. • Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow #J-18808-Ljbffr ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands‐on experience with AWS data services Amazon Redshift or comparable cloud data‐warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data‐modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

expertise in writing complex, optimized SQL queries for data extraction and transformation. ETL/ELT: Hands-on experience building data pipelines using frameworks like Apache Spark, Apache Airflow, DBT, Fivetran, Matillion, or equivalent. Databases: Solid understanding of relational databases (e.g., PostgreSQL, MySQL) and experience with modern data ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

Platform. Experience with data modelling, schema design, and data architecture best practices. Working knowledge of Infrastructure as Code (IaC) tools and practices. Experience with Apache Spark, Apache Airflow, or equivalent data processing and orchestration frameworks. Experience supporting machine learning, analytics, or AI-focused data platforms. Familiarity with ...

storage architectures. Strong understanding of data processing performance and optimisation. Experience with some of the following data frameworks and infrastructure technologies is highly desirable: Apache Spark, Apache Airflow, Kafka, and Elasticsearch or OpenSearch. Experience with relevant database technologies is highly desirable, including PostgreSQL, MongoDB, and vector databases ...

Knowledge of: MLOps Feature Engineering Exposure to data analysis and ML tools such as: scikit-learn pandas NumPy XGBoost LightGBM Familiarity with orchestration tools: Apache Oozie Apache Airflow Why you’ll love it here We want to make Prima a happy and empowering place to work. ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

knowledge of optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow … cost optimisation strategies for cloud data platforms Experience with data quality frameworks and implementation Experience with data visualisation tools like Power BI or Apache Superset Experience with other cloud data platforms like AWS, GCP or Oracle Experience with modern unified data platforms like Databricks or Microsoft Fabric Experience with ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

stores). Hands-on experience building, scaling, and shipping large-scale data platforms in production. Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop). Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis). End-to-end exposure … data pipeline lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster). Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg). Advanced SQL skills and proficiency in Python. Strong background ...

platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation … quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well-governed ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business … Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow or similar tools Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems Preferred qualifications, capabilities, and skills Data modeling ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business … Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow or similar tools Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems Preferred qualifications, capabilities, and skills Data modeling ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business … Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow or similar tools Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems Preferred qualifications, capabilities, and skills Data modeling ...

Azure infrastructure design, deployment and management, ideally in an NHS setting Knowledge and experience of contributing and building complex ETL/ELT pipelines-airflow, data build tool, Fivetran, Matillion, Azure Data Factory Knowledge and experience of contributing to the integration of dozens of source systems Knowledge and experience ...