1 to 25 of 253 Apache Airflow Jobs in London

Business Intelligence - Data EngineerExperience working with SEI frameworks and Data Delivery to various firms using Snowflake, Airflow, Azure Data Factory and Microsoft Azure.Day to Day job Duties: (what this person will do on a daily/weekly basis)Work closely with client stakeholders to understand business needs, translate them … sprint planning, and review meetings to ensure continuous communication and progress tracking.Design, build, and optimize scalable data pipelines using Azure Data Factory and Apache Airflow for scheduling and orchestration.Implement efficient data ingestion, transformation, and movement processes from various sources into Snowflake Data Warehouse.Develop reusable components and templates ...

Business Intelligence - Data EngineerExperience working with SEI frameworks and Data Delivery to various firms using Snowflake, Airflow, Azure Data Factory and Microsoft Azure. Day to Day job Duties: (what this person will do on a daily/weekly basis)Work closely with client stakeholders to understand business needs, translate … sprint planning, and review meetings to ensure continuous communication and progress tracking. Design, build, and optimize scalable data pipelines using Azure Data Factory and Apache Airflow for scheduling and orchestration. Implement efficient data ingestion, transformation, and movement processes from various sources into Snowflake Data Warehouse. Develop reusable components ...

Linux operating systems, including command-line administration and troubleshooting. Solid experience with SQL or data pipeline technologies, Experience with data processing frameworks such as Apache Airflow, Apache Flink or Google Dataflow, with a practical understanding of ETL pipeline design and the ability to diagnose issues across data …/IP networking principles, including the ability to diagnose connectivity issues and interpret network traffic. Familiarity with event streaming platforms such as Apache Kafka, including an understanding of producer/consumer patterns and how message queues are used to move data reliably between systems at scale. Familiarity with infrastructure ...

London --------------------------------------- JOB DETAILS Role Title: Data Engineer (Python Enterprise Developer) Must have skills: -Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow -Preferred Domain: Energy trading Minimum number of relevant years of experience: 8+ Detailed Job Description: JD: • 8+ years of experience in python … communication skills with an ability to lead a team and keep them motivated. • Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow #J-18808-Ljbffr ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production.Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline lifecycle … development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices ...

platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation … quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well-governed ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence … Iceberg, Hudi, or similar)Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt)Experience designing and operating orchestration pipelines using Airflow or similar toolsExperience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systemsDemonstrated experience using enterprise-authorized AI capabilities within ...

storage, compute and managed data services.Strong experience with modern data warehousing/lakehouse tools (e.g., Snowflake, BigQuery, Redshift, Databricks).Hands-on orchestration experience (e.g., Airflow, Dagster, Prefect) and CI/CD for data pipelines.Experience with streaming and event-driven data (e.g., Kafka, Kinesis, Pub/Sub) is highly desirable.Proficiency ...

Experience with cloud platforms such as AWS, Azure, or GCP. Knowledge of Data Warehouse and Data Lake architectures. Experience with orchestration tools such as Airflow, ADF, or equivalent. Understanding of data governance, data quality, and security practices. Familiarity with CI/CD pipelines and version control tools. Preferred Skills ...

transformation, orchestration, reverse ETL. You've worked with tools like DLT/Fivetran/Airbyte (ingestion), BigQuery/Snowflake/Redshift (warehouse), DBT (transformation), Airflow/similar (orchestration).Strong Python programming - you write clean, testable, maintainable code with solid error handling and logging.Fluent SQL - you can write complex queries ...

technical specifications, and client-ready deliverables.Desirable Experience with advanced analytical techniques, such as machine learning, predictive modelling, or optimisation.Familiarity with data orchestration tools (e.g., Airflow, dbt, Prefect, Cloud Composer).Experience with containerisation or infrastructure-as-code (e.g., Docker, Kubernetes, Terraform).Knowledge of data warehousing concepts and technologies.Experience contributing ...

React, Node) using Webpack, Typescript, and GitHub.Experience with Docker and/or Kubernetes and containerized deployments (AKS).Experience with workflow orchestration tools such as Apache Airflow or n8n.Experience of code quality, test, CI/CD and package managers such as Jest, Lint, Brigade, and Helm.Experience with Kafka ...

using Webpack, Typescript, and GitHub. Experience with Docker and/or Kubernetes and containerised deployments (AKS). Experience with workflow orchestration tools such as Apache Airflow or n8n. Experience of code quality, test, CI/CD and package managers such as Jest, Lint, Brigade, and Helm. Experience with ...

Synapse/DatabricksAWS Glue/Redshift/EMRGoogle BigQuery/DataflowEnsure systems are secure, scalable, and cost-efficientData Pipeline & Workflow ManagementDevelop orchestration workflows (e.g., Airflow, Prefect, Azure Data Factory)Monitor and troubleshoot pipelines to ensure high availability and performanceImplement robust error handling, logging, and alertingData Quality & GovernanceEnsure data quality ...

comfortable triaging, escalating, and driving incidents to resolution under pressure.What you'll do:Design, develop, and maintain high-performance, product-centric data pipelines using Airflow, DBT, and Python.Architect and optimize the massive-scale data warehouse and lakehouse that serves as our single source of truth for all customer data ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark.Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.Implement continuous integration and continuous deployment ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration ...

/CD, DevOps, and Infrastructure as Code practices for data platforms. Metadata management, lineage tooling, and data observability/monitoring experience. Familiarity with dbt, Apache Airflow, Azure Data Factory, Kafka, Event Hubs, or comparable orchestration/integration tools. Familiarity with enterprise AI platforms — Azure AI, AWS SageMaker, Google ...

frameworks, orchestration tooling, and data engineering best practices. Familiarity with technologies such as Spark, Python, SQL, Delta Lake, Unity Catalog, dbt, Azure Data Factory, Airflow, Kafka, or equivalent tooling. Relevant certifications such as Snowflake SnowPro, Databricks Certified Data Engineer, or cloud platform certifications would be an advantage. Skills: Strong ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow.Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time streams … preferred), including data modelling and design skills.Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data.Proven experience with Apache Airflow for data orchestration and scheduling at scale.Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must.Definition and design ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal ...

Data Engineer Associate) Experience with streaming data (Kinesis, MSK/Kafka) Familiarity with Infrastructure as Code (Terraform/CloudFormation) Experience with orchestration tools (Airflow, dbt) Experience in financial domain OtherLanguages English: C2 Proficient Seniority Senior London, United Kingdom of Great Britain and Northern Ireland Req. VR-124691 Strategy & Design ...