126 to 150 of 153 Apache Airflow Jobs in London

third-party data from APIs into scalable storage layers • Developing dbt transformations and ELT workflows for analytics and machine learning • Orchestrating workloads using Airflow and modern CI/CD practices • Supporting model execution environments, including Azure ML Studio (experience not required) What you’ll be doing • Designing and building … scalable data pipelines in a modern Azure environment • Developing modular, production-grade ELT workflows (dbt, Airflow, SQL, Python) • Modelling data for analytics, BI, forecasting, and machine learning use cases • Optimising data architectures for performance, cost, and reliability • Working closely with data scientists, software engineers, and investment teams • Troubleshooting ...

third-party data from APIs into scalable storage layers • Developing dbt transformations and ELT workflows for analytics and machine learning • Orchestrating workloads using Airflow and modern CI/CD practices • Supporting model execution environments, including Azure ML Studio (experience not required) What you’ll be doing • Designing and building … scalable data pipelines in a modern Azure environment • Developing modular, production-grade ELT workflows (dbt, Airflow, SQL, Python) • Modelling data for analytics, BI, forecasting, and machine learning use cases • Optimising data architectures for performance, cost, and reliability • Working closely with data scientists, software engineers, and investment teams • Troubleshooting ...

e.g. AWS, GCP, Azure), infrastructure-as-code, and scalable architecture patterns Hands-on experience with SQL, Python, and modern data integration frameworks such as Apache Kafka, airflow or dbt Proven ability to deliver enterprise-grade customer data solutions, including customer data platforms and or API based integration Excellent ...

e.g. AWS, GCP, Azure), infrastructure-as-code, and scalable architecture patterns Hands-on experience with SQL, Python, and modern data integration frameworks such as Apache Kafka, airflow or dbt Proven ability to deliver enterprise-grade customer data solutions, including customer data platforms and or API based integration Excellent ...

team to make sure our data function not only delivers but also drives strategic decision-making. Our Tech Stack: Python, Docker, Dagster, dbt, Fivetran, Apache Iceberg, Snowflake, S3, Glue, ECS, and Omni. We're constantly evolving our stack and welcome input from engineering leaders on how we can improve … scaling data platforms in a high-growth or start-up environment. Strong expertise in Python and SQL, with deep experience in orchestration frameworks (Dagster, Airflow, Prefect). Advanced knowledge of data modelling and architecture (Kimball dimensional modelling, Data Vault etc). Hands-on experience with dbt, modern data warehouses ...

team to make sure our data function not only delivers but also drives strategic decision-making. Our Tech Stack: Python, Docker, Dagster, dbt, Fivetran, Apache Iceberg, Snowflake, S3, Glue, ECS, and Omni. We're constantly evolving our stack and welcome input from engineering leaders on how we can improve … scaling data platforms in a high-growth or start-up environment. Strong expertise in Python and SQL, with deep experience in orchestration frameworks (Dagster, Airflow, Prefect). Advanced knowledge of data modelling and architecture (Kimball dimensional modelling, Data Vault etc). Hands-on experience with dbt, modern data warehouses ...

schemas to support business requirements Develop and maintain data ingestion and processing systems using various tools and technologies, such as SQL, NoSQL, ETL, Luigi, Airflow, Argo, etc. Implement data storage solutions using different types of databases, such as relational, non-relational, or cloud-based. Working collaboratively with the client …/Azure SQL, PostgreSQL) You have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving ...

schemas to support business requirements Develop and maintain data ingestion and processing systems using various tools and technologies, such as SQL, NoSQL, ETL, Luigi, Airflow, Argo, etc. Implement data storage solutions using different types of databases, such as relational, non-relational, or cloud-based. Working collaboratively with the client …/Azure SQL, PostgreSQL) You have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving ...

have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI … Django, Airflow, Kafka, ETL, CI/CD, experience with some or all would be beneficial but you'll pick it up on the job What's in it for you: Salary to £100k + bonus Health Insurance Life Assurance 25 days holiday Training and self-development budget Impactful role ...

Nice-to-Have Skills: Experience with GCP/BigQuery or other cloud data warehouses (e.g., Snowflake, Redshift). Familiarity with data orchestration tools (e.g., Airflow). Experience with data visualisation platforms (such as Preset.io/Apache Superset or other). Exposure to CI/CD pipelines, ideally using ...

job. You will be taking ownership of a stack built on: Ingestion: Weld Warehousing & ML: BigQuery and Vertex AI Transformation: dbt Orchestration: Airflow BI: Looker Studio What This Role Involves This is a Full Stack Analytics Engineering role. You will own the lifecycle of data from the API source … Meta, Google Ads, Shopify, GA4, and other sources into BigQuery. Orchestration: You will manage the scheduling, dependencies, and monitoring of our data workflows using Airflow , ensuring data arrives on time and in the correct sequence. Custom Ingestion: You will write custom Python scripts/extractors to fetch data where ...

job. You will be taking ownership of a stack built on: Ingestion: Weld Warehousing & ML: BigQuery and Vertex AI Transformation: dbt Orchestration: Airflow BI: Looker Studio What This Role Involves This is a Full Stack Analytics Engineering role. You will own the lifecycle of data from the API source … Meta, Google Ads, Shopify, GA4, and other sources into BigQuery. Orchestration: You will manage the scheduling, dependencies, and monitoring of our data workflows using Airflow , ensuring data arrives on time and in the correct sequence. Custom Ingestion: You will write custom Python scripts/extractors to fetch data where ...

models for ML, insights, and search pipelines Design production-ready data pipelines delivering high-quality datasets to partner teams Work with Databricks, AWS, dbt, Airflow, and Looker to implement robust solutions Lead technically complex initiatives and contribute to the team’s roadmap Improve engineering efficiency through internal tooling … Requirements Proven track record delivering production-ready data engineering projects end-to-end Strong SQL and Python skills, with experience on platforms like Databricks, Airflow, dbt, and Looker Experience in data modelling, data warehousing, and performance/cost optimisation Skilled in stakeholder management, documentation, and project planning Experience developing ...

models for ML, insights, and search pipelines Design production-ready data pipelines delivering high-quality datasets to partner teams Work with Databricks, AWS, dbt, Airflow, and Looker to implement robust solutions Lead technically complex initiatives and contribute to the team’s roadmap Improve engineering efficiency through internal tooling … Requirements Proven track record delivering production-ready data engineering projects end-to-end Strong SQL and Python skills, with experience on platforms like Databricks, Airflow, dbt, and Looker Experience in data modelling, data warehousing, and performance/cost optimisation Skilled in stakeholder management, documentation, and project planning Experience developing ...

models for ML, insights, and search pipelines Design production-ready data pipelines delivering high-quality datasets to partner teams Work with Databricks, AWS, dbt, Airflow, and Looker to implement robust solutions Lead technically complex initiatives and contribute to the team's roadmap Improve engineering efficiency through internal tooling … Requirements Proven track record delivering production-ready data engineering projects end-to-end Strong SQL and Python skills, with experience on platforms like Databricks, Airflow, dbt, and Looker Experience in data modelling, data warehousing, and performance/cost optimisation Skilled in stakeholder management, documentation, and project planning Experience developing ...

fast-paced, challenging environment. Key Responsibilities: Design, implement, and debug ETL pipelines to process and manage complex datasets. Leverage big data tools, including Apache Kafka, Spark, and Airflow, to deliver scalable solutions. Collaborate with stakeholders to ensure data quality and alignment with business goals. Utilize programming expertise … data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like API Gateway, Lambda, Redshift, Glue, Cloudwatch, etc. ...

fast-paced, challenging environment. Key Responsibilities: Design, implement, and debug ETL pipelines to process and manage complex datasets. Leverage big data tools, including Apache Kafka, Spark, and Airflow, to deliver scalable solutions. Collaborate with stakeholders to ensure data quality and alignment with business goals. Utilize programming expertise … data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like API Gateway, Lambda, Redshift, Glue, Cloudwatch, etc. ...

fast-paced, challenging environment. Key Responsibilities: Design, implement, and debug ETL pipelines to process and manage complex datasets. Leverage big data tools, including Apache Kafka, Spark, and Airflow, to deliver scalable solutions. Collaborate with stakeholders to ensure data quality and alignment with business goals. Utilize programming expertise … data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like API Gateway, Lambda, Redshift, Glue, Cloudwatch, etc. ...

fast-paced, challenging environment. Key Responsibilities: Design, implement, and debug ETL pipelines to process and manage complex datasets. Leverage big data tools, including Apache Kafka, Spark, and Airflow, to deliver scalable solutions. Collaborate with stakeholders to ensure data quality and alignment with business goals. Utilize programming expertise … data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like API Gateway, Lambda, Redshift, Glue, Cloudwatch, etc. ...

Lead Data Engineer - Python, SQL, DBT, Airflow, BigQuery, AWS/GCP OB are partnered with a UK FinTech Unicorn, undergoing huge growth within their Data function, and are therefore looking for a Lead Data Engineer to join their Data team. This is a key role, where … variety of stakeholders across the business, working on several exciting greenfield projects and scaling their data systems. Lead Data Engineer - Python, SQL, DBT, Airflow, BigQuery, AWS/GCP Key skills and experience: 3+ years of experience in a Leadership role Python/SQL DBT, BigQuery, Airflow AWS/ ...

Data Engineering Lead - Python, SQL, DBT, Airflow, BigQuery, AWS/GCP OB are partnered with a UK FinTech Unicorn, undergoing huge growth within their Data function, and are therefore looking for a Lead Data Engineer to join their Data team. This is a key role, where … variety of stakeholders across the business, working on several exciting greenfield projects and scaling their data systems. Data Engineering Lead - Python, SQL, DBT, Airflow, BigQuery, AWS/GCP Key skills and experience: 3+ years of experience in a Leadership role Python/SQL DBT, BigQuery, Airflow AWS/ ...

Data Engineering Lead - Python, SQL, DBT, Airflow, BigQuery, AWS/GCP OB are partnered with a UK FinTech Unicorn, undergoing huge growth within their Data function, and are therefore looking for a Lead Data Engineer to join their Data team. This is a key role, where … variety of stakeholders across the business, working on several exciting greenfield projects and scaling their data systems. Data Engineering Lead - Python, SQL, DBT, Airflow, BigQuery, AWS/GCP Key skills and experience: 3+ years of experience in a Leadership role Python/SQL DBT, BigQuery, Airflow AWS/ ...

setting. MSc in Computer Science, Machine Learning, or a related field. Experience building data pipelines (realtime or batch) & data quality using modern toolchain (e.g., Apache Spark, Kafka, Airflow, dbt). Strong foundational knowledge of machine learning and deep learning algorithms, including deep neural networks, supervised/unsupervised learning ...

setting. MSc in Computer Science, Machine Learning, or a related field. Experience building data pipelines (realtime or batch) & data quality using modern toolchain (e.g., Apache Spark, Kafka, Airflow, dbt). Strong foundational knowledge of machine learning and deep learning algorithms, including deep neural networks, supervised/unsupervised learning ...

across the business. Act as an expert in SQL across technologies like Postgres, Cloud SQL, BigQuery, and Redshift . Build and orchestrate pipelines using Airflow . Support IaC and CI/CD best practices via GitHub. Work closely with data engineers, analysts, data scientists, and product teams. About … Strong experience building analytics-focused pipelines. Proficient with dbt , SQL, and cloud warehouse technologies. Knowledge of Kimball modelling and testing frameworks. Experience with Airflow or similar orchestrators. Understanding of IaC and CI/CD principles. Collaborative and comfortable working with cross-functional teams. The client is looking to move ...