101 to 125 of 130 Apache Airflow Jobs

language used for automation (e.g. Python, JavaScript, TypeScript). · Experience integrating automated tests into CI/CD pipelines, or using orchestration platforms such as Apache Airflow · Practical experience working with real and/or virtual devices (Android, iOS, emulators/simulators) and dealing with the challenges of scale ...

right questions, and making reasonable bets. Bonus Experience with IoT/telemetry, energy data, or time-series workloads Familiarity with event-driven architectures, GraphQL, Airflow, or Python Personal interest in energy, climate tech, or automation Track record of owning complex systems end-to-end 5+ years of experience This ...

grade Python services, batch and streaming pipelines, and scheduled workflows Build production-grade data pipelines – develop idempotent, observable, and cost-efficient pipelines using BigQuery, Airflow, Google Workflows, and Cloud Run Implement CI/CD for ML workloads – automated validation, monitoring, rollback strategies, and model lifecycle management Establish reliability … productionizing ML models in batch and/or real-time environments Hands-on experience with analytical data warehouses (BigQuery or equivalent) and workflow orchestration (Airflow or similar) Experience deploying ML systems on GCP, including Cloud Run, GCS, and IAM Infrastructure-as-Code experience (Terraform or equivalent) Experience with feature ...

Prepare and manage data across multiple platforms with a focus on accuracy Build and maintain BI dashboards using tools such as QuickSight, DBT, and Airflow Support audit, due diligence, and financial reporting requirements Your Skills and Experience Strong commercial experience in reporting, analytics, and financial analysis Advanced SQL capability ...

influence stakeholders across multiple regions. Technical Environment Experience across some of the following technologies is advantageous: AIops and enterprise automation frameworks Workflow orchestration platforms (Airflow, Argo, Prefect) Configuration and automation tools (Ansible, Salt, Puppet) ITSM and ITOM platforms Event-driven automation and runbook-as-code approaches Messaging technologies such ...

Skills (Bonus Points) Experience with MLOps tooling (MLflow, Kubeflow, Weights & Biases, or similar). Building data pipelines (real‐time or batch) using tools like Apache Spark, Kafka, Airflow, or dbt. Familiarity with recommender systems, transformers, or graph neural networks. Exposure to model optimisation techniques (quantisation, distillation, efficient inference ...

monitoring, and CI/CD automation. Build container-oriented ML platforms (EKS-first) while evaluating alternative orchestration tools with similar capabilities (Kubeflow, SageMaker, MLflow, Airflow, etc.). Implement hybrid MLOps + LLMOps workflows, including prompt/version governance, evaluation frameworks, and monitoring for LLM-based systems. Serve … CodeBuild/CodePipeline or equivalent). Hands-on experience with at least one major MLOps toolset and awareness of alternatives: MLflow, Kubeflow, SageMaker Pipelines, Airflow, BentoML, KServe, Seldon. Deep understanding of model lifecycle management (feature engineering->training → registry → deployment → monitoring). Experience implementing or supporting LLMOps pipelines, including: prompt ...

Lead Data Engineer (Databricks & Cloud Migration) £650 per day (Inside IR35) Initial 6-Month Contract London – Hybrid Working (3 Days per Week Onsite) We are partnering with a leading international bank on a major Databricks ...

checks. Operational or production support mindset; experience in stabilizing live environments. Hands-on use of scheduling or workflow tools such as GitLab CI, Jenkins, Airflow, or Control M (or similar). Strong proficiency working with Linux platforms. Experience managing SSH servers. Ability to write and maintain Ansible scripts ...

Concurrency, Preemption, Stress testing frameworks Testbench Architecture and Implementation GFX API Exposure : Vulkan/DX11/DX12 Exposure Scripting and automation skills (Python, Make, Airflow etc) Embedded FW Development and Debugging Benchmarking and Performance Analysis Windows and/or Linux OS Kernel Architecture C/C++, GNU Toolchain, Visual ...

fast-paced, challenging environment. Key Responsibilities: Design, implement, and debug ETL pipelines to process and manage complex datasets. Leverage big data tools, including Apache Kafka, Spark, and Airflow, to deliver scalable solutions. Collaborate with stakeholders to ensure data quality and alignment with business goals. Utilise programming expertise … data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like API Gateway, Lambda, Redshift, Glue, Cloudwatch, etc. ...

workloads. Practical experience working with AWS cloud services, including compute, storage, and security tooling. Infrastructure-as-code experience using Terraform. Distributed data processing using Apache Spark. Workflow scheduling and orchestration with Apache Airflow. Containerisation and environment management using Docker. CI/CD pipeline implementation and maintenance using GitLab. ...

integration failures, data inconsistencies, and production issues Key Skills Python & SQL (production-grade pipeline development) REST APIs, webhooks, OAuth, event-driven architecture Orchestration tools: Airflow, Prefect, or Dagster Cloud platforms: AWS, GCP, or Azure Docker & Kubernetes Microsoft 365 & Microsoft Copilot Desirable Skills Vector databases and embedding pipelines Real-time … robust, production-grade data pipelines from scratch. Deep familiarity with integration patterns: REST APIs, webhooks, OAuth, and event-driven architectures. Experience with orchestration tools (Airflow, Prefect, or Dagster) and transformation frameworks (dbt or similar). Comfortable across cloud platforms (AWS, GCP, or Azure) and with containerisation (Docker, Kubernetes). ...

integration failures, data inconsistencies, and production issues Key Skills Python & SQL (production-grade pipeline development) REST APIs, webhooks, OAuth, event-driven architecture Orchestration tools: Airflow, Prefect, or Dagster Cloud platforms: AWS, GCP, or Azure Docker & Kubernetes Microsoft 365 & Microsoft Copilot Desirable Skills Vector databases and embedding pipelines Real-time … robust, production-grade data pipelines from scratch. Deep familiarity with integration patterns: REST APIs, webhooks, OAuth, and event-driven architectures. Experience with orchestration tools (Airflow, Prefect, or Dagster) and transformation frameworks (dbt or similar). Comfortable across cloud platforms (AWS, GCP, or Azure) and with containerisation (Docker, Kubernetes). ...

code review, source control, build, test, deploy, and operations Awesome If You: Are experienced in Rust/Java/Kotlin Have experience with AWS, Apache Kafka, Kafka Streams, Apache Beam/Flink/Spark – especially deployment, monitoring & debugging Have experience with productisation of Machine Learning research projects … familiar with Airflow or other workflow orchestration tools, and worked with Kubernetes Understand data lake systems and file formats like Parquet, Orc, Athena Have some relevant AWS or Kafka certifications A vibrant, diverse company pushing ourselves and the technology to deliver beyond the cutting edge A team of motivated ...

Technology, Engineering, or associated discipline Experience working with advanced ETL data workflows including technologies such as AWS EMR, Azure Synapse, Azure Data Factory, or Apache Hive/Spark/Airflow Supervisory experience Experience with IaC deployment of AKS/EKS/GKE architecture is highly desired Experience with ...

engineering, or a related field, and has a foundational understanding of databases, programming, and data processing concepts. Your activities include: Operational Health: Monitor daily Airflow DAGs and Airbyte syncs; act as the first line of defense for pipeline failures. DBT Operations: Monitor daily DBT jobs, troubleshoot model failures … master student in the direction of Data Engineering You are fluent in English Strong SQL skills are a must; Python for general scripting and Airflow logic. Intermediate understanding of data modeling concepts, of how tables relate (Joins, Primary Keys, and basic Star Schema concepts). Ability to read logs ...

Design, build, and maintain scalable data pipelines and workflows to support analytics, fraud detection, model development, and ongoing data monitoring (e.g., using Spark, Airflow, or similar distributed systems). Leverage and build agentic AI and LLM-powered systems to automate data exploration, anomaly detection, vendor evaluation, and investigative workflows … both batch and near‐real‐time use cases. Experience building and maintaining data pipelines and workflows in distributed or large‐scale environments (e.g., Spark, Airflow, Databricks, or similar technologies). Demonstrated ability to evaluate and work with third‐party data vendors or external datasets, including designing tests for data ...

We're looking for an experienced Analytics Engineer to help shape our data foundations and deliver high quality, trusted insights across the business. You'll design and maintain scalable analytics models, ensure data quality, and ...

Manchester, UK, to develop scalable APIs and manage big data solutions. Candidates should have 8+ years in software or data engineering, with skills in Apache Spark, Airflow, and a strong foundation in Java, Python, or Scala. The role requires collaboration with cross-functional teams to deliver high-quality ...

high-impact, early-stage business £100K plus equity in a well-funded Series A start-up. Own and shape a GCP data platform (BigQuery, Airflow, dbt). Real progression into a future leadership role. We’re working with an early-stage, well-funded tech-led start-up. They … production-ready systems that scientists can trust and use daily. You’ll manage and extend a GCP-first platform, working with tools like BigQuery, Airflow, dbt, and and so much more. Sound like you? What experience you’ll need to apply Strong Data Engineering experience in a cloud environment ...

reliable, production-ready systems that scientists can trust and use daily.You’ll manage and extend a GCP-first platform, working with tools like BigQuery, Airflow, dbt, and Terraform. You’ll also help build internal applications using Python, FastAPI, and React, giving teams easy ways to trigger workflows and access … backend services and working with data workflows A few years of experience with GCP, including BigQuery and cloud-native tools Hands-on experience with Airflow, dbt, and modern data pipelines Infrastructure as code experience, ideally using Terraform Experience building or contributing to internal tools or application Strong ...

framework on top of our data platform. Implement MCP- and A2A-based architectures for agent collaboration and orchestration. Integrate AI services with data pipelines (Airflow, Spark, Trino, StarRocks, DBT). Build APIs, SDKs, and tooling to enable other teams to use the framework. Work with platform and application teams … Have) Strong programming skills in Python with 5+ years of experience Proven experience with data engineering or distributed systems (Spark, Hive, Trino, StarRocks, DBT, Airflow, Kubernetes). Deep understanding of LLM-based systems and agent frameworks (LangChain, LlamaIndex, DSPy, OpenAI MCP, A2A). Hands-on experience with API design ...

Principal Data Engineer Location: Newcastle/Manchester/Leeds/Southwest England | Hybrid with regular client-site travel Salary: £100,000 - £140,000 Security Clearance: Eligible for UK SC clearance required Employer Overview Our client ...

Senior Data Engineer | Python | Airbyte | Apache | BigQuery | EducationTech | Remote, London | £70k Looking to help shape Data Strategy for a scaling Tech-for-Good business I'm partnered with a AI-first data and analytics platform built for schools, multi-academy trusts and education groups. Their technology connects fragmented school … Engineer to build and maintain the data infrastructure that underpins the entire platform.You’ll own and extend the ELT pipeline - Airbyte (OSS) for extraction, Airflow on Astro for orchestration, dbt via Cosmos for transformation - and the BigQuery lakehouse beneath it, with a semantic layer on top.These are hands ...