126 to 147 of 147 Apache Airflow Jobs in England

grade Python services, batch and streaming pipelines, and scheduled workflows Build production-grade data pipelines – develop idempotent, observable, and cost-efficient pipelines using BigQuery, Airflow, Google Workflows, and Cloud Run Implement CI/CD for ML workloads – automated validation, monitoring, rollback strategies, and model lifecycle management Establish reliability … productionizing ML models in batch and/or real-time environments Hands-on experience with analytical data warehouses (BigQuery or equivalent) and workflow orchestration (Airflow or similar) Experience deploying ML systems on GCP, including Cloud Run, GCS, and IAM Infrastructure-as-Code experience (Terraform or equivalent) Experience with feature ...

data flow, including supporting delivery in air-gapped and other restricted environments. Delivering data applications in an automated and continuous fashion. Utilising tools like Airflow for orchestration and automating data pipeline deployments using DevOps tools and techniques (Docker, Containers, Terraform, Kubernetes) Building data pipelines using both cloud-native … experience as a DataOps Engineer delivering DataOps solutions in public sector or similarly regulated environments. Developing, orchestrating and maintaining data pipelines using tools like Airflow, Prefect, or Dagster. Containerising data applications using Docker and deploying them to Container Platforms (EKS, AKS and Kubernetes). Implementing CI/CD pipelines ...

than one-off scripts or analysis tooling. Strong Python development skills, with experience building performant, maintainable applications. Experience with workflow orchestration tools such as Airflow, Dagster, or Prefect. Understanding of retries, dependency management, idempotency, backfills, and operational recovery. Experience with analytical or columnar databases such as ClickHouse or similar ...

influence stakeholders across multiple regions. Technical Environment Experience across some of the following technologies is advantageous: AIops and enterprise automation frameworks Workflow orchestration platforms (Airflow, Argo, Prefect) Configuration and automation tools (Ansible, Salt, Puppet) ITSM and ITOM platforms Event-driven automation and runbook-as-code approaches Messaging technologies such ...

building and leading a team of engineers working in a related system. Experience with build, release and deployment infrastructure technologies such as Spinnaker, Jenkins, Airflow, Docker, Kubernetes, Terraform, Bazel, etc. Ability to attract, hire, and coach engineers who meet the Databricks hiring standards – can up‐level existing team ...

building and leading a team of engineers working in a related system. Experience with build, release and deployment infrastructure technologies such as Spinnaker, Jenkins, Airflow, Docker, Kubernetes, Terraform, Bazel, etc. Ability to attract, hire, and coach engineers who meet the Databricks hiring standards – can up‐level existing team ...

clear, performant, and reliable queries to transform large complex datasets) You have experience building and owning automated data pipelines with tools like DBT/Airflow Experimentation & Statistics: Experience running A/B tests and analysing the results to drive product decisions Ownership: You enjoy working independently in a fast ...

your initiative and get on with resolving it! From a tech perspective, it would be most excellent if you've come from an AWS, Apache AirFlow, Python, Spark type of environment and understand these technologies (or alternatives) and what they do. You'll still be considered ...

Develop batch and ELT/ETL data workflows for ingesting and processing large datasets Work with technologies such as Python, Java, Spark, SQL, and Airflow Apply good software engineering practices, including testing, documentation, and CI/CD Contribute to cloud‐based systems (AWS), focusing on reliability, monitoring, and automation … data engineering concepts such as data lakes, batch processing, and distributed systems Experience building or operating data pipelines using tools like Spark and Airflow Familiarity with cloud platforms (AWS preferred) and production monitoring Clear communication skills and a collaborative approach to problem‐solving. Why join us? Join our team ...

integration failures, data inconsistencies, and production issues Key Skills Python & SQL (production-grade pipeline development) REST APIs, webhooks, OAuth, event-driven architecture Orchestration tools: Airflow, Prefect, or Dagster Cloud platforms: AWS, GCP, or Azure Docker & Kubernetes Microsoft 365 & Microsoft Copilot Desirable Skills Vector databases and embedding pipelines Real-time … robust, production-grade data pipelines from scratch. Deep familiarity with integration patterns: REST APIs, webhooks, OAuth, and event-driven architectures. Experience with orchestration tools (Airflow, Prefect, or Dagster) and transformation frameworks (dbt or similar). Comfortable across cloud platforms (AWS, GCP, or Azure) and with containerisation (Docker, Kubernetes). ...

integration failures, data inconsistencies, and production issues Key Skills Python & SQL (production-grade pipeline development) REST APIs, webhooks, OAuth, event-driven architecture Orchestration tools: Airflow, Prefect, or Dagster Cloud platforms: AWS, GCP, or Azure Docker & Kubernetes Microsoft 365 & Microsoft Copilot Desirable Skills Vector databases and embedding pipelines Real-time … robust, production-grade data pipelines from scratch. Deep familiarity with integration patterns: REST APIs, webhooks, OAuth, and event-driven architectures. Experience with orchestration tools (Airflow, Prefect, or Dagster) and transformation frameworks (dbt or similar). Comfortable across cloud platforms (AWS, GCP, or Azure) and with containerisation (Docker, Kubernetes). ...

focus areas: Building scalable data pipelines using Apache Spark & Airflow Working within a cloud-based, containerised environment (Kubernetes) Contributing to a wider data platform modernisation programme Supporting both batch and streaming workloads Tech: Python, Java, Spark, Airflow, SQL, Cloud (AWS/Azure), Kubernetes (Kafka/streaming nice ...

highthroughput, faulttolerant ingestion pipelines for batch and streaming data landing in BigQuery. • Lead data ingestion architecture patterns using Cloud Storage (GCS), Dataflow, Dataproc, Composer (Airflow), Pub/Sub, BigQuery Storage Write API, and related services. Define data loading frameworks, mapping rules, schema evolution strategy, and metadata management. Create reusable … performance bottlenecks, and crossplatform data integration issues. Top 10 Skillset & Qualities (Ideal Candidate) 1. Deep Expertise in Google Cloud Data Services BigQuery, GCS, Dataflow (Apache Beam), Pub/Sub, Dataproc, Cloud Composer, Storage Write API. 2. Data Ingestion Engineering Mastery Handson experience designing frameworks to load data from APIs ...

Location: UK Based/Hybrid Contract: 6 Months Initial Rate: £600-700 per day Start: ASAP We’re looking for a highly competent Snowflake Data Consultant to support a major cloud data transformation programme. This ...

account management, so your focus stays exactly where it belongs, on hunting and closing. You will sell a respected range of air containment, airflow and efficiency solutions that help data centres cut costs, improve performance and reduce their carbon footprint, backed by a technical team that knows the detail. ...

high-impact, early-stage business £100K plus equity in a well-funded Series A start-up. Own and shape a GCP data platform (BigQuery, Airflow, dbt). Real progression into a future leadership role. We’re working with an early-stage, well-funded tech-led start-up. They … production-ready systems that scientists can trust and use daily. You’ll manage and extend a GCP-first platform, working with tools like BigQuery, Airflow, dbt, and and so much more. Sound like you? What experience you’ll need to apply Strong Data Engineering experience in a cloud environment ...

Principal Data Engineer Location: Newcastle/Manchester/Leeds/Southwest England | Hybrid with regular client-site travel Salary: £100,000 - £140,000 Security Clearance: Eligible for UK SC clearance required Employer Overview Our client ...

within air-gapped environments, ensuring secure deployment of ML solutions Collaborate with data scientists to operationalise ML models Utilise tools such as MLflow and Airflow for model tracking and orchestration Support cloud-based deployments across AWS and Google Cloud Platform Ensure performance, reliability, and scalability of machine learning systems … Experience building CI/CD pipelines using GitLab (YAML) Understanding of DevSecOps principles and tooling Exposure to ML lifecycle tools such as MLflow and Airflow Ability to work in highly secure or air-gapped environments Strong problem-solving and collaboration skills What you'll get in return Opportunity ...

distributed processing frameworks such as Spark or Databricks Build and optimise data workflows across cloud platforms (AWS, Azure, GCP) Implement orchestration frameworks such as Airflow Test, troubleshoot, and improve data products to ensure reliability and quality Collaborate with engineering teams to deliver robust, maintainable systems Key Requirements: Experience building … Experience working with at least some of: Distributed computing frameworks (e.g. Spark, Databricks) Cloud platforms (AWS, Azure, or GCP) Workflow orchestration tools (e.g. Airflow) Strong understanding of data system reliability, testing, and performance Experience troubleshooting and improving production data systems Strong problem-solving skills and attention to detail Ability ...

distributed processing frameworks such as Spark or Databricks Build and optimise data workflows across cloud platforms (AWS, Azure, GCP) Implement orchestration frameworks such as Airflow Test, troubleshoot, and improve data products to ensure reliability and quality Collaborate with engineering teams to deliver robust, maintainable systems Key Requirements: Experience building … Experience working with at least some of: Distributed computing frameworks (e.g. Spark, Databricks) Cloud platforms (AWS, Azure, or GCP) Workflow orchestration tools (e.g. Airflow) Strong understanding of data system reliability, testing, and performance Experience troubleshooting and improving production data systems Strong problem-solving skills and attention to detail Ability ...

Job Title Data Engineering Lead Salary £115k + Equity Company Description 9fin.com - Series B AI-powered fintech SaaS Job Description As Data Engineering Lead, you will own the technical direction of a platform centralising data ...

A leading fintech firm in London seeks a Data Engineering Lead to own the technical direction of a platform transforming data for the debt capital market. You'll guide the architecture while building a team ...