1 to 25 of 41 Apache Airflow Jobs in London

create the queries and processes necessary to move everything to the new. As migration work wraps up, the role will transition to leveraging Airflow and other tooling to automate operational and financial processes. This is a great role for exceptionally outgoing individuals as all work will require extensive collaboration … design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process ...

create the queries and processes necessary to move everything to the new. As migration work wraps up, the role will transition to leveraging Airflow and other tooling to automate operational and financial processes. This is a great role for exceptionally outgoing individuals as all work will require extensive collaboration … design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

privacy Eligibility for UK SC clearance Nice to have (but not essential) Exposure to AWS, Azure or GCP Familiarity with tools such as Airflow, dbt, Spark Awareness of CI/CD pipelines or containerisation Experience in public sector or regulated environments Important note for applicants This role is deliberately ...

governance, security, and regulatory compliance . Ability to work closely with technical and business teams to deliver impactful solutions. Nice-to-Haves: Orchestration tools (Airflow, dbt, Prefect) or streaming frameworks (Kafka, Kinesis). CI/CD experience and DevOps practices for data workflows. Exposure to data visualisation platforms (PowerBI ...

/GCP beneficial) Strong data modelling knowledge Understanding of data governance and security best practices Ability to translate business needs into scalable solutions Desirable Airflow, dbt, Kafka, CI/CD, BI tools, governance platforms, financial services experience. Interested in building modern, enterprise-scale data platforms in a highly regulated ...

stream data processing. Cloud Pub/Sub: For real-time messaging and event ingestion. Cloud Storage: For scalable and durable object storage. Cloud Composer (Apache Airflow): For workflow orchestration and pipeline management. Cloud Dataproc: For managed Hadoop and Spark clusters (good to have). Cloud SQL/Cloud ...

driving innovation in data engineering. Desirable Criteria Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. Familiarity with Apache Airflow, data modelling, and metadata management Experience influencing enterprise data strategy and contributing to architectural governance. To apply for this position please submit ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

Skills and Experience * Strong commercial experience working with AWS in production environments. * Proficiency in Python and SQL for data processing and orchestration. * Experience with Airflow for workflow management. * Knowledge of cloud data warehousing, ideally Redshift. * Experience with infrastructure as code and CI/CD tooling such as Terraform ...

systems for unstructured data retrieval Exposure to cloud infrastructure (AWS, GCP, or Azure) for scalable data pipeline deployment Experience with modern data orchestration tools (Airflow, Prefect, dbt) What We Offer Direct access to senior football operations leadership, with genuine influence over recruitment decisions A greenfield build the opportunity ...

familiarity with data preparation, transformation, visualisation, and cloud deployment. Experience with Google Cloud Platform and tools such as Google Pub/Sub, Dataflow, Dataproc, Airflow, Dataprep, Data Fusion. Experience with CI/CD and DevOps tooling (e.g., Jenkins, Kubernetes). Experience working within Agile delivery models (Scrum, Kanban), including ...

Data Engineer within GCP environments Hands-on experience with BigQuery, Dataflow and Spanner Strong programming capability in Python, Java or Scala Experience with Apache Beam/Dataflow and distributed processing frameworks Experience designing and managing ETL/ELT pipelines Solid understanding of data modelling and database design Experience with … workflow orchestration tools such as Apache Airflow Strong understanding of data governance, security and scalability Desirable Skills for the GCP Data Engineer: Experience with Pub/Sub, Cloud Composer or Cloud Data Fusion Exposure to CI/CD, DevOps and Infrastructure as Code Experience with real-time streaming ...

experience. Hands-on experience with Hadoop - configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

Proven experience in a Lead/Senior Data Engineering role Strong handson experience with Snowflake Expertise in ETL/ELT and orchestration tools (e.g. Airflow, dbt, Matillion) Strong SQL and Python skills Experience building scalable data platforms in cloud environments Previous experience leading or mentoring engineers Strong communication ...

discipline Solid Python programming skills Good working knowledge of SQL Comfortable using Git for version control Desirables: Exposure to workflow orchestration tools (e.g. Prefect, Airflow, Dagster) Experience with cloud data warehouses (Azure SQL, Snowflake) or dbt Basic familiarity with Docker and BI tools (Power BI, Tableau) Interest in shipping ...

pipeline design Mastery of data modeling techniques Skilled in data ingestion, integration, and transformation using modern tools (Data Factory, Databricks, SQL DB, Synapse, Glue, Airflow, Kinesis, Redshift) Hands-on experience with DevOps tooling: GitHub Actions, Azure DevOps, SonarQube, PyTest Proven ability to coordinate incident, problem, and change management processes ...

projects are approached and ensuring delivery standards are upheld Designing and governing scalable pipelines using BigQuery, Dataflow, Datastream, Data Fusion, Dataproc and Cloud Composer (Airflow) Leading best practice across dbt, Terraform, Docker and CI/CD Driving Data Mesh adoption, domain ownership, and interoperable data products Setting modelling standards ...

projects are approached and ensuring delivery standards are upheld Designing and governing scalable pipelines using BigQuery, Dataflow, Datastream, Data Fusion, Dataproc and Cloud Composer (Airflow) Leading best practice across dbt, Terraform, Docker and CI/CD Driving Data Mesh adoption, domain ownership, and interoperable data products Setting modelling standards ...

What they're looking for Strong Python + data engineering fundamentals Experience building scalable data pipelines/platforms Exposure to distributed systems (Kafka, Spark, Airflow etc.) Interest in AI/ML systems is a plus ...

peer to peer feedback and evolving healthy, curious and humble teams. Tech Stack: Python, Javascript/Typescript, React/React Native, AWS, GraphQL, Snowflake, Airflow, DDD. This is an incredible opportunity for a Senior Software Engineer to join a unique company as they embark on a period of significant ...

healthcare, fintech, or other regulated environments Exposure to LLM infrastructure, model serving, or agentic AI frameworks in production Familiarity with data platforms (BigQuery, Airflow, data pipelines) Experience with CI/CD pipelines (GitHub Actions, Cloud Build) and policy-as-code tools Exposure to observability stacks (OpenTelemetry, Prometheus, Cloud Monitoring ...

production environments; Python expertise - idiomatic, well-tested, production-grade code, not notebook scripts * ETL/ELT pipeline design and implementation at scale; orchestration with Airflow, Prefect, or equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics * Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics … documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building and maintaining ...

tools including Intelligent Document Processing and LLMs. Required: Strong programming skills i.e Python. Experience working with any front end languages ideally React. Experience with Airflow, Kubernetes, Docker, SQL/NoSQL databases. Must have strong financial serivces background/open to fintech startup firms. Cloud and CI/CD knowledge. ...

right questions, and making reasonable bets. Bonus Experience with IoT/telemetry, energy data, or time-series workloads Familiarity with event-driven architectures, GraphQL, Airflow, or Python Personal interest in energy, climate tech, or automation Track record of owning complex systems end-to-end 5+ years of experience This ...