1 to 25 of 26 Apache Airflow Jobs in the City of London

embedding models, and RAG architectures. Familiarity with document processing pipelines, chunking strategies, and semantic search patterns. Familiarity with data pipeline and orchestration tools (e.g., Airflow, dbt, Spark, Kafka, cloud-native ETL tools) and batch vs. streaming patterns. Understanding of data governance (catalog, lineage, security, RBAC, masking, compliance requirements like ...

driving innovation in data engineering. Desirable Criteria Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. Familiarity with: - Apache Airflow - Data modelling and metadata management Experience influencing enterprise data strategy and contributing to architectural governance. To apply for this position please submit ...

Skills & Experience Proven experience as a Data Engineer , BI Engineer , or similar role. Strong proficiency in SQL , Python , and modern data engineering frameworks (e.g., Airflow, dbt, Spark, etc.). Experience with cloud platforms such as AWS, Azure, or GCP. Solid understanding of data warehousing and ETL/ELT architecture. ...

data models and map various data sources to support downstream analytics. Pipeline Engineering: Build scalable ETL pipelines and data quality solutions using tools like Airflow, Luigi, or Azkaban. Codification: Translate intricate business logic into high-quality code using Python, Scala, or Java. Innovation: Evaluate and implement new technologies across ...

Excellence Lead the evolution of the enterprise data landscape using modern platforms such as Databricks, Snowflake, Azure Synapse, and BigQuery. Oversee data engineering with Apache Airflow, dbt, and Prefect, ensuring performance, governance, and alignment with enterprise metadata standards (Collibra, Alation, Microsoft Purview). Drive adoption of Delta Lake ...

/CD for data pipelines. Good to have skills Although not essential, the following skills are desired by the client: Kafka/Kinesis, Airflow, CodePipeline. BI tools (Power BI/Tableau). Docker/OpenShift; metadata driven pipelines. 3-8+ years Data Engineering experience. Cloud data engineering ...

Experienced building real pipelines — not just notebooks Curious about how financial systems actually work Bonus points for: Exposure to ML workflows Familiarity with dbt, Airflow, Docker Experience handling financial or semi-structured data Interest in LLM infrastructure and vector databases Finance background is not required. Intellectual horsepower and ownership ...

Experience working with complex, multi source data environments, ideally including ecommerce and subscription data. Experience with data pipeline orchestrations tools such as Cloud Composer, Airflow or equivalent. Comfort working in fast moving environments with imperfect starting points. Ability to balance best practice with pragmatism and delivery speed. Strong communication ...

Tech Stack Language: Expert-level Python and advanced SQL . Cloud: Extensive experience with GCP (BigQuery, Cloud Storage, Dataflow/PubSub, Cloud Composer/Airflow). Tools: Experience with data orchestration tools and version control (Git). Environment: Proficiency in building production-grade pipelines. What We’re Looking ...

technologies . You will be utilising and playing with tools across the entire data ecosystem. 🚀Essential tech skills include proficiency in Python, Spark, and Airflow , alongside experience with major data platforms such as Snowflake, Databricks, and BigQuery . 🚀Familiarity with data transformation and infrastructure as code tools like ...

Greenfield project. What you'll need to succeed Strong Python and SQL are crucial to this role On-prem to Snowflake migration experience Airflow and DBT experience GitHub experience as you will be building CI/CD pipelines as part of this role Any experience within post-trade ...

platforms in a cloud-native environment (AWS, GCP, or Azure). Deep understanding of modern data stacks (e.g. Snowflake/BigQuery/Redshift, dbt, Airflow, BI tools. Comfort operating at both strategic and hands-on levels. Track record of partnering with non-technical stakeholders and influencing at exec level. ...

/lending/risk/portfolio datasets Time series familiarity (trend/seasonality, cohorts over time, monitoring) Experience with data tools such as dbt, Airflow, Snowflake/BigQuery/Databricks (or similar) Experience creating automated data quality frameworks or reconciliation checks Benefits: £45,000 – £55,000 salary Hybrid working ...

mindset, with openness to feedback and diverse perspectives. Bonus points if you have the following. Experience building and maintaining production-grade data pipelines (e.g. Airflow, Dagster) or integrating data from APIs, web scraping, or third-party providers. Familiarity with cloud environments such as Google Cloud or AWS, and deploying ...

Skills & Experience Strong experience with Python and/or SQL in a production environment. Hands-on experience with modern data engineering frameworks such as: Apache Spark, Databricks Airflow/Prefect/Dagster Kafka/EventHub/Kinesis Experience building data solutions on at least one major cloud platform ...

Scala experience Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms (LLMs, predictive models ...

Cloud Run, GKE, CloudSQL, BigQuery) Bare Metal Large datasets & geospatial data (PostGIS a plus) Infrastructure Terraform/Terragrunt GitHub Actions CI/CD Kubernetes Airflow/Prefect/SQLMesh Grafana What We’re Looking For 4+ years building production systems Strong React + Python experience Advanced SQL (performance tuning ...

candidate has: Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms (LLMs, predictive models ...

/Gas), who is technically well versed in systematic trading development, and analysis data-sets (time series, market-data). Stack: Python, AWS, ETL, Airflow The team has a flat structure, where developers are interfaced with PMs and traders, to ensure the software systems created are bespoke. ...

drive business impact through data Technical Environment Languages: Python, SQL, Bash Cloud: Azure, AWS Data & Platforms: Snowflake, Delta Lake, Redis, Azure Data Lake Tools: Airflow, DBT, Docker Infrastructure & Operations: Terraform, GitHub Actions, Azure DevOps, Azure Monitor Desirable Skills Experience with data platforms such as Snowflake or Azure Data Lake … Experience deploying models as APIs (e.g. FastAPI, Azure Functions) Knowledge of monitoring, model performance tracking and observability practices Familiarity with orchestration tools such as Airflow or Azure Data Factory ...

What you'll be doing: Design, develop, and maintain scalable data architectures and ETL pipelines Build and manage data models and data warehouse solutions (Airflow, dbt, and Redshift) Write clean, efficient Python and SQL code for data processing and transformation Integrate data from internal and third-party APIs … principles Ability to optimise data pipelines for performance and maintainability Strong collaboration skills and a proactive, problem-solving mindset Bonus Points For Experience with Airflow and/or dbt Experience working in Agile environments (Scrum/Kanban) Exposure to DevOps practices or CI/CD pipelines You'll join ...

ingestion workflows Implement data quality frameworks and enforce data contracts Build CI/CD pipelines and automated testing for data workflows Orchestrate pipelines using Airflow, Prefect, or Dagster Improve observability, monitoring, and lineage across data systems Mentor junior engineers and elevate engineering standards Key Skills and Experience Management … Data Engineering experience Expert-level SQL and advanced data modelling Strong Python Experience with Snowflake and Databricks Cloud experience Experience with orchestration tools (Airflow, Prefect) CI/CD and DataOps experience for data platforms This role will suit someone who is looking to remain hands on , whilst leading ...

Group) I'm hiring a Mid-Level Data Engineer (3+ years’ experience) to help scale and optimise a robust data platform built on Python, Airflow, dbt and Redshift. 🔧 What you’ll be doing: • Scaling and optimising ELT pipelines (Python + Airflow) • Enhancing the data warehouse (performance, cost, complexity … looking for: • 3+ years in Data Engineering (FinTech/startup/scale-up ideal) • Strong Python + SQL expertise • Production-grade workflow orchestration (Airflow/Dagster/Prefect) • Deep understanding of data warehousing + ELT principles • Interest in Open Banking & payments This is a pivotal role at a high ...

Technical Lead/Data Architect Start date: ASAP Duration: 6 months Location: London, flexible working *Please note. - You must be registered as a freelance contractor (Ltd Co, Sole Trader) or work via an umbrella company ...

Senior Data Engineer AI SaaS | Data as a Product | London (Hybrid) This is a brilliant opportunity for someone who loves solving problems at the intersection of data and product and wants real ownership in a ...