1 to 25 of 66 Apache Airflow Jobs in London

create the queries and processes necessary to move everything to the new. As migration work wraps up, the role will transition to leveraging Airflow and other tooling to automate operational and financial processes. This is a great role for exceptionally outgoing individuals as all work will require extensive collaboration … design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process ...

create the queries and processes necessary to move everything to the new. As migration work wraps up, the role will transition to leveraging Airflow and other tooling to automate operational and financial processes. This is a great role for exceptionally outgoing individuals as all work will require extensive collaboration … design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

governance, security, and regulatory compliance . Ability to work closely with technical and business teams to deliver impactful solutions. Nice-to-Haves: Orchestration tools (Airflow, dbt, Prefect) or streaming frameworks (Kafka, Kinesis). CI/CD experience and DevOps practices for data workflows. Exposure to data visualisation platforms (PowerBI ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud Storage … . Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

scalable data solutions. Good understanding of data modelling, database design and normalisation. Proficiency in Python and SQL, ideally with experience of frameworks such as Airflow, TensorFlow or Spark. Willingness to build basic–intermediate backend skills (e.g. Python/Django) to support data integrations. Familiarity with data versioning, data quality … pipelines in production. Exposure to data lakes, data warehousing and modern data storage patterns. Experience with cloud platforms (AWS/Azure) and tools like Apache Airflow, Terraform or SageMaker. ...

/GCP beneficial) Strong data modelling knowledge Understanding of data governance and security best practices Ability to translate business needs into scalable solutions Desirable Airflow, dbt, Kafka, CI/CD, BI tools, governance platforms, financial services experience. Interested in building modern, enterprise-scale data platforms in a highly regulated ...

environment Proven experience delivering at least one production-grade data platform Strong hands-on capability with modern data engineering tools such as Python, SQL, Airflow, Spark, Kafka, dbt, Snowflake (Essential) Experience with backend technologies (e.g. Java/Spring Boot) Experiencewith cloud platforms (AWS, Azure, or GCP) and cloud-native ...

stream data processing. Cloud Pub/Sub: For real-time messaging and event ingestion. Cloud Storage: For scalable and durable object storage. Cloud Composer (Apache Airflow): For workflow orchestration and pipeline management. Cloud Dataproc: For managed Hadoop and Spark clusters (good to have). Cloud SQL/Cloud ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

driving innovation in data engineering. Desirable Criteria Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. Familiarity with Apache Airflow, data modelling, and metadata management Experience influencing enterprise data strategy and contributing to architectural governance. To apply for this position please submit ...

Skills and Experience * Strong commercial experience working with AWS in production environments. * Proficiency in Python and SQL for data processing and orchestration. * Experience with Airflow for workflow management. * Knowledge of cloud data warehousing, ideally Redshift. * Experience with infrastructure as code and CI/CD tooling such as Terraform ...

Ability to create clean, structured datasets for reporting and analytics Comfortable working in a fast-paced, evolving data environment Nice to Have Experience with Airflow or Dagster Working with JSON/semi-structured data Familiarity with CI/CD pipelines and Git Why Join Work with high-volume, real ...

building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps , CI/ ...

across ML Ops, data architecture and data engineering best practices and scalable data solutions. Proficiency in Python and SQL, ideally with frameworks such as Airflow, PyTorch or Spark. Good understanding of supervised and unsupervised learning and how to choose/apply the right model to a business problem. Familiarity ...

cloud data platforms (AWS preferred) Deep experience with AWS, Databricks, Power Platform, and Redshift (Snowflake a plus) Proficiency in AWS Glue, Qlik Talend, DBT, Airflow, and modern data integration tools. Excellent knowledge of Python, SQL, PowerQuery (M), and preferably Scala or PySpark Working knowledge with enterprise architecture frameworks (e.g. ...

performance marketing data Experience working on large-scale data transformation or platform modernisation projects Tech Stack Warehouse: Snowflake Transformation: dbt BI: Looker Orchestration: Airflow Version Control: GitHub Other Tools: Adverity, OpenMetadata, Jira, Confluence Benefits Hybrid working (minimum of 3 days per week in office) 25 days annual leave plus ...

Proven experience in a Lead/Senior Data Engineering role Strong handson experience with Snowflake Expertise in ETL/ELT and orchestration tools (e.g. Airflow, dbt, Matillion) Strong SQL and Python skills Experience building scalable data platforms in cloud environments Previous experience leading or mentoring engineers Strong communication ...

data/AI platforms Deep knowledge of GCP (BigQuery, Dataflow, Pub/Sub, Vertex AI) Experience with LLMs (e.g. Gemini or similar) Familiarity with Airflow/Composer and dbt Understanding of data governance and financial services regulation Strong stakeholder management skills What’s on Offer Competitive salary + bonus ...

familiarity with data preparation, transformation, visualisation, and cloud deployment. Experience with Google Cloud Platform and tools such as Google Pub/Sub, Dataflow, Dataproc, Airflow, Dataprep, Data Fusion. Experience with CI/CD and DevOps tooling (e.g., Jenkins, Kubernetes). Experience working within Agile delivery models (Scrum, Kanban), including ...

/lending/risk/portfolio datasets Time series familiarity (trend/seasonality, cohorts over time, monitoring) Experience with data tools such as dbt, Airflow, Snowflake/BigQuery/Databricks (or similar) Experience creating automated data quality frameworks or reconciliation checks Benefits: £45,000 – £55,000 salary Hybrid working ...

experience. Hands-on experience with Hadoop - configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

discipline Solid Python programming skills Good working knowledge of SQL Comfortable using Git for version control Desirables: Exposure to workflow orchestration tools (e.g. Prefect, Airflow, Dagster) Experience with cloud data warehouses (Azure SQL, Snowflake) or dbt Basic familiarity with Docker and BI tools (Power BI, Tableau) Interest in shipping ...

pipeline design Mastery of data modeling techniques Skilled in data ingestion, integration, and transformation using modern tools (Data Factory, Databricks, SQL DB, Synapse, Glue, Airflow, Kinesis, Redshift) Hands-on experience with DevOps tooling: GitHub Actions, Azure DevOps, SonarQube, PyTest Proven ability to coordinate incident, problem, and change management processes ...

deployment, ensuring scalable, high-quality, and governed data platforms built on Google Cloud. Key skills: Experience in GCP Data Engineering - expertise in BigQuery and Airflow Experience with Infrastructure as Code (Terraform) Data quality frameworks and validation techniques CI/CD pipeline development and solution deployment Exposure to AI/ ...