1 to 25 of 58 Apache Airflow Jobs in London

systems are developed and used responsibly and ethically. Tooling the Future: Get hands-on with cutting-edge technologies like Hugging Face, PyTorch, TensorFlow, Apache Spark, Apache Airflow, and other modern data and ML frameworks. Collaborate and Lead: Partner closely with ML Engineers, Data Scientists, and Researchers … services. Programming Prowess: Strong programming skills in Python and SQL are essential. Big Data Ecosystem Expertise: Hands-on experience with big data technologies like Apache Spark, Kafka, and data orchestration tools such as Apache Airflow or Prefect. ML Data Acumen: Solid understanding of data requirements for machine ...

Platform DevOps team to build and run highly reliable CI/CD pipelines and infrastructure for the bank’s modern data estate (AWS + Airflow + Databricks). Key Responsibilities Design, build and maintain Infrastructure as Code using CloudFormation and AWS services Manage and scale Apache Airflow … call rotation and incident resolution Essential Skills & Experience 5+ years as a DevOps/Platform Engineer Strong AWS experience + CloudFormation Production experience with Apache Airflow (MWAA) Lambda , Linux + Bash scripting CI/CD expertise (Git, Jenkins, GitHub Actions) IaC and GitOps practices Nice to Have Terraform ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud Storage … . Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

governance, security, and regulatory compliance . Ability to work closely with technical and business teams to deliver impactful solutions. Nice-to-Haves: Orchestration tools (Airflow, dbt, Prefect) or streaming frameworks (Kafka, Kinesis). CI/CD experience and DevOps practices for data workflows. Exposure to data visualisation platforms (PowerBI ...

architecture to support data consumption for reporting systems, operational applications, and analytics platforms. Implement automation, orchestration, and scalable pipeline frameworks using tools such as Apache Airflow. Collaborate with business stakeholders and technical teams to align the data platform with enterprise data strategy. Support hospitality analytics use cases, including guest … predictive analytics, recommendation engines, and customer behaviour analytics. Ability to design AI-ready data platforms for future analytics use cases. Preferred Skills Experience with Apache Airflow for pipeline orchestration. Knowledge of CI/CD pipelines, DevOps, and Git-based development workflows. Experience with data governance, metadata management ...

data models and map various data sources to support downstream analytics. Pipeline Engineering: Build scalable ETL pipelines and data quality solutions using tools like Airflow, Luigi, or Azkaban. Codification: Translate intricate business logic into high-quality code using Python, Scala, or Java. Innovation: Evaluate and implement new technologies across ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

Skills and Experience * Strong commercial experience working with AWS in production environments. * Proficiency in Python and SQL for data processing and orchestration. * Experience with Airflow for workflow management. * Knowledge of cloud data warehousing, ideally Redshift. * Experience with infrastructure as code and CI/CD tooling such as Terraform ...

working with modern cloud technologies. 🔧 Key Skills & Technologies 🐍 Python (NumPy, Pandas, Polars, BeautifulSoup, Selenium, Requests) 🗄️ PostgreSQL/Advanced SQL ⚡ Azure Databricks ☁️ AWS (S3, Lambda) 🔄 Apache Airflow 🔧 Git & CI/CD (Azure DevOps, Jenkins) 🧠 Key Responsibilities Build & maintain Python-based ETL pipelines Work on data ingestion, transformation & processing Handle ...

data quality, reliability, and performance optimisation Support secure and compliant data practices Requirements 5+ years Data Engineering experience Strong Python & SQL skills Experience with Airflow, Dagster, or similar orchestration tools Cloud experience (GCP) Strong understanding of data modelling & ETL/ELT GCP stack (BigQuery, Composer, Dataproc, etc.) Data governance ...

with SQL and modern data warehousing technologies (e.g., Snowflake, BigQuery, Redshift, Databricks). Proven experience building production-ready pipelines using tools such as dbt, Airflow, Dagster, Prefect, or similar. Experience integrating data from HRIS and talent systems (e.g., Workday, BambooHR, Greenhouse, Lever, HiBob, Personio). Ability to work with ...

with dimensional modelling, star schemas, SCDs etc.Experience within regulated financial servicesDesirableExperience in pensions, asset management or institutional investingUnderstanding of performance measurement and attributionExposure to Airflow or dbtCloud platform familiarity (Azure or AWS)Knowledge of data governance best practiceAbout YouHighly detail-driven with a strong approach to data qualityComfortable operating ...

1classification ) Strong SQL&Python Hands-onETL/ELTexperience(e.g.,Matillion,Talend,Fivetran,ADF) Experiencewith Snowflake (AWS/Azure/GCPbeneficial) Strongdatamodellingknowledge Understandingofdatagovernanceandsecuritybestpractices Abilitytotranslatebusinessneedsintoscalablesolutions Desirable Airflow,dbt,Kafka,CI/CD,BItools,governanceplatforms,financialservicesexperience. Interestedinbuildingmodern,enterprise-scaledataplatformsinahighlyregulatedenvironment Applynow ...

experience Time-series experience (market data, telemetry, pricing, events) Streaming exposure (Kafka/Event Hubs/Kinesis) Experience with Temporal (or similar orchestrators like Airflow/Dagster/Prefect) Any exposure to AI agents/automation tooling 👉 Apply ...

cloud data platforms (AWS preferred) Deep experience with AWS, Databricks, Power Platform, and Redshift (Snowflake a plus) Proficiency in AWS Glue, Qlik Talend, DBT, Airflow, and modern data integration tools. Excellent knowledge of Python, SQL, PowerQuery (M), and preferably Scala or PySpark Working knowledge with enterprise architecture frameworks (e.g. ...

performance marketing data Experience working on large-scale data transformation or platform modernisation projects Tech Stack Warehouse: Snowflake Transformation: dbt BI: Looker Orchestration: Airflow Version Control: GitHub Other Tools: Adverity, OpenMetadata, Jira, Confluence Benefits Hybrid working (minimum of 3 days per week in office) 25 days annual leave plus ...

Proven experience in a Lead/Senior Data Engineering role Strong handson experience with Snowflake Expertise in ETL/ELT and orchestration tools (e.g. Airflow, dbt, Matillion) Strong SQL and Python skills Experience building scalable data platforms in cloud environments Previous experience leading or mentoring engineers Strong communication ...

data/AI platforms Deep knowledge of GCP (BigQuery, Dataflow, Pub/Sub, Vertex AI) Experience with LLMs (e.g. Gemini or similar) Familiarity with Airflow/Composer and dbt Understanding of data governance and financial services regulation Strong stakeholder management skills What’s on Offer Competitive salary + bonus ...

statistics, and modelling • Experience building production-grade ML systems (e.g. forecasting, optimisation, CV) • Solid ML Ops and data engineering experience • Proficiency in Python & SQL (Airflow, PyTorch, Spark a plus) • Experience with AWS (SageMaker, Lambda, etc.) • Interest in backend development (Django or Go) • High level STEM Degree from a leading ...

cross-functional teams. Strong stakeholder communication skills at all levels. Comfortable leading iterative delivery using Agile practices. Exposure to tools such as dbt, Airflow/ADF/Dagster, Snowflake/Databricks/Fabric, and metadata/governance systems is beneficial. Ready to Lead in a Role with Real Influence ...

/lending/risk/portfolio datasets Time series familiarity (trend/seasonality, cohorts over time, monitoring) Experience with data tools such as dbt, Airflow, Snowflake/BigQuery/Databricks (or similar) Experience creating automated data quality frameworks or reconciliation checks Benefits: £45,000 – £55,000 salary Hybrid working ...

Tech Stack Language: Expert-level Python and advanced SQL . Cloud: Extensive experience with GCP (BigQuery, Cloud Storage, Dataflow/PubSub, Cloud Composer/Airflow). Tools: Experience with data orchestration tools and version control (Git). Environment: Proficiency in building production-grade pipelines. What We’re Looking ...

pipeline design Mastery of data modeling techniques Skilled in data ingestion, integration, and transformation using modern tools (Data Factory, Databricks, SQL DB, Synapse, Glue, Airflow, Kinesis, Redshift) Hands-on experience with DevOps tooling: GitHub Actions, Azure DevOps, SonarQube, PyTest Proven ability to coordinate incident, problem, and change management processes ...

ReplacingMergeTree, cluster management, query optimisation). Experience with probabilistic entity resolution or record linkage (Splink, Dedupe, Zingg, or custom implementations). Familiarity with Dagster, Airflow, Prefect, or similar orchestration frameworks. Experience with DuckDB or other embedded analytical engines. Background in B2B data, company data, or master data management. Experience ...