1 to 25 of 223 Permanent Apache Airflow Jobs in the UK

bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than … class technical solutions to customer and prospect business problems. In this role, you will be exposed to a wide range of use cases where Apache Airflow sits at the center of the solution. The impacts you will make will enable customers to rapidly grow their businesses by adopting ...

golden record creation, and distribution to downstream consumers across the firm. We operate on a modern cloud‐native stack centered on Snowflake , AWS , and Apache Airflow , and follow engineering best practices including CI/CD, code review, and automated testing. Key Responsibilities Design, build, and maintain scalable data … pipelines for Security Master, Account Master, and Counterparty Master using Python and Apache Airflow . Develop and optimize complex data transformations, stored procedures, and views in Snowflake , ensuring high performance and data quality. Own the end‐to‐end lifecycle of reference data — from source ingestion and normalization through ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

Description:Job Title Senior Airflow/Cloud Engineer (Workload Automation)Location BirminghamCorporate Title Assistant Vice PresidentService Automation provides an integrated set of management tools for IT professionals to programmatically provision and automate the operational maintenance of their hybrid application environment. It focuses on the following key capabilities:Technology Operations … responsible for availability, latency, performance, efficiency, change management, monitoring, automation, emergency response, and capacity planning for the centralized platforms of Control-M, Automic, Airflow as well as Google services Cloud Composer, Workflows and Scheduler which are utilized across the Bank for workload automation/job scheduling/batch ...

software development team Hands‐on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands‐on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … highly scalable data engineering solutions that power Moody’s next‐generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moody’s technical ...

Develop and maintain data pipelines using Python, SQL, and Apache Airflow. • Containerise data applications and workflows using Docker. • Build and optimise data solutions using AWS services including: • Redshift, OpenSearch, Lambda, Glue, Step Functions, and Batch. • Collaborate with cross-functional teams to deliver robust, secure, and scalable data infrastructure. • Manage … improvement initiatives. • Ensure data quality, governance, and security best practices are followed. Requirements Strong programming skills in Python and proficiency in SQL. Experience with Apache Airflow for DAG orchestration and monitoring. Hands-on experience with Docker for containerisation. Proficient in AWS data services: Redshift, OpenSearch, Lambda, Glue, Step ...

Data Engineer - Newcastle upon Tyne (Hybrid- 3 days onsite) - SQL, Airflow, Python, Power BI - Future Microsoft Fabric Transformation Project - Up to £55,000 Are you a Data Engineer looking for an opportunity where you can influence the direction of a growing data platform rather than simply maintain existing processes … data capabilities and embarking on an exciting data transformation journey. This role offers the chance to work hands-on with Azure SQL, Airflow, Python and Power BI while playing a key role in the organisation's future migration to Microsoft Fabric. You'll join a small, highly visible team ...

London --------------------------------------- JOB DETAILS Role Title: Data Engineer (Python Enterprise Developer) Must have skills: -Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow -Preferred Domain: Energy trading Minimum number of relevant years of experience: 8+ Detailed Job Description: JD: • 8+ years of experience in python … communication skills with an ability to lead a team and keep them motivated. • Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow #J-18808-Ljbffr ...

bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than … Astronomer lets businesses do more with their data. To learn more, visit www.astronomer.io. About this role: At Astronomer, we’re redefining how companies run Apache Airflow at scale. Our R&D organization is home to some of the most innovative minds in cloud infrastructure and open-source software. ...

software and site reliability engineering. Preferred: Experience with distributed ML frameworks such as Horovod or TensorFlow Distributed. Familiarity with data engineering pipelines such as Apache Airflow or Apache Spark. Knowledge of chaos engineering tools and compliance frameworks such as GDPR, SOC 2, or ISO 27001. Relocation benefits ...

3+ years of experience with Spark, Databricks, Snowflake, Hadoop, or other big data technologies . Nice to Have: Experience with orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Glue. Knowledge of streaming technologies such as Kafka, Kinesis, or Pub/Sub. Experience with data warehouse ...

delivering enterprise-scale data platform solutions. A strong background in Solution Architecture or Technical Architecture. Hands-on experience with open-source technologies such as Apache Spark, Trino, Airflow, Apache Superset, or similar data engineering platforms. Experience working with cloud platforms including Azure, AWS, or GCP. Experience producing ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

Knowledge of data governance, metadata management, and data quality frameworks Hands-on experience with technologies such as Azure Data Factory, Databricks, Snowflake, Microsoft Fabric, Airflow, or similar Excellent stakeholder management and communication skills Experience mentoring engineers and providing technical leadership Strong analytical and problem-solving abilities Experience working within ...

stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Experience working with and designing infrastructures for large‐scale data processing (Hive, Snowflake, NoSQL databases). Experience with data governance ...

warehousing concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical ...

agile SDLC environment, leveraging modern programming languages, DevOps, and test-driven development Exposure to Docker, Git hands on experience with pipeline orchestration tools like Apache Airflow, Autosys, Control-M – Preferred Apache Ariflow proficient at working with large and complex code bases using GitLab/Github, Fork/ ...

live entertainment. What You'll Be Doing Build & Deliver Data Solutions Design, build and maintain scalable ETL/ELT pipelines using tools such as Airflow and dbt. Build and manage both real-time and batch data pipelines that support AXS products and platforms. Data Modelling & Analytics Develop and optimise … with cloud data warehouse technologies including Snowflake, Redshift or BigQuery. Data Pipeline & Streaming Experience Experience with dbt for data transformation and modelling. Knowledge of Apache Airflow or similar orchestration tools such as Prefect or Dagster. Experience with streaming platforms such as Kafka, StreamSets, AWS Kinesis or Flink. Engineering ...

relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL. Orchestration Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect. Data Quality and Lineage Familiarity with tools and techniques for data testing, documentation and lineage. Examples include Great Expectations or dbt. ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

Kanban. Strong analytical and problem‐solving skills, with a focus on delivering high-quality solutions. Experience with modern data engineering tools such as Airflow, dbt, Dataform, or similar technologies. Excellent communication and stakeholder management skills, with the ability to explain technical concepts to non‐technical audiences. What we offer ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments CI/CD experience with a focus on production ownership and reliability ...

etc.). Working knowledge of Kubernetes for container orchestration. Hands‐on experience with several of the following: ETL/ELT orchestration and pipeline tooling (Airflow, Meltano, DBT, etc.) Real‐time streaming platforms (Kafka, Redpanda, Active/RabbitMQ, etc.) SQL databases, particularly MySQL Desired Experience Experience with data warehousing tools ...

Spark, Databricks, Hadoop or cloud-native alternatives). Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow). Strong experience with data engineering concepts (ETL/ELT, data integration, warehousing, vector databases/search) A little bit more about us... ...