1 to 25 of 219 Apache Airflow Jobs in London

Business Intelligence - Data EngineerExperience working with SEI frameworks and Data Delivery to various firms using Snowflake, Airflow, Azure Data Factory and Microsoft Azure.Day to Day job Duties: (what this person will do on a daily/weekly basis)Work closely with client stakeholders to understand business needs, translate them … sprint planning, and review meetings to ensure continuous communication and progress tracking.Design, build, and optimize scalable data pipelines using Azure Data Factory and Apache Airflow for scheduling and orchestration.Implement efficient data ingestion, transformation, and movement processes from various sources into Snowflake Data Warehouse.Develop reusable components and templates ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

Linux operating systems, including command-line administration and troubleshooting. Solid experience with SQL or data pipeline technologies, Experience with data processing frameworks such as Apache Airflow, Apache Flink or Google Dataflow, with a practical understanding of ETL pipeline design and the ability to diagnose issues across data …/IP networking principles, including the ability to diagnose connectivity issues and interpret network traffic. Familiarity with event streaming platforms such as Apache Kafka, including an understanding of producer/consumer patterns and how message queues are used to move data reliably between systems at scale. Familiarity with infrastructure ...

London --------------------------------------- JOB DETAILS Role Title: Data Engineer (Python Enterprise Developer) Must have skills: -Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow -Preferred Domain: Energy trading Minimum number of relevant years of experience: 8+ Detailed Job Description: JD: • 8+ years of experience in python … communication skills with an ability to lead a team and keep them motivated. • Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow #J-18808-Ljbffr ...

databases utilised in modern AI applications. Strong Python development skills. Data & Infrastructure: Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure: Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases: PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Ecosystem: Pandas Polars ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production.Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline lifecycle … development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence … Iceberg, Hudi, or similar)Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt)Experience designing and operating orchestration pipelines using Airflow or similar toolsExperience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systemsDemonstrated experience using enterprise-authorized AI capabilities within ...

storage, compute and managed data services.Strong experience with modern data warehousing/lakehouse tools (e.g., Snowflake, BigQuery, Redshift, Databricks).Hands-on orchestration experience (e.g., Airflow, Dagster, Prefect) and CI/CD for data pipelines.Experience with streaming and event-driven data (e.g., Kafka, Kinesis, Pub/Sub) is highly desirable.Proficiency ...

transformation, orchestration, reverse ETL. You've worked with tools like DLT/Fivetran/Airbyte (ingestion), BigQuery/Snowflake/Redshift (warehouse), DBT (transformation), Airflow/similar (orchestration).Strong Python programming - you write clean, testable, maintainable code with solid error handling and logging.Fluent SQL - you can write complex queries ...

working with Azure or AWS cloud platforms Experience with modern data platforms such as Databricks, Snowflake or Redshift Experience using orchestration tools such as Airflow, Azure Data Factory or Prefect Strong understanding of data warehousing and data modelling principles Excellent problem-solving skills with a delivery-focused mindset Comfortable ...

Experience with cloud platforms such as AWS, Azure, or GCP. Knowledge of Data Warehouse and Data Lake architectures. Experience with orchestration tools such as Airflow, ADF, or equivalent. Understanding of data governance, data quality, and security practices. Familiarity with CI/CD pipelines and version control tools. Preferred Skills ...

Synapse/DatabricksAWS Glue/Redshift/EMRGoogle BigQuery/DataflowEnsure systems are secure, scalable, and cost-efficientData Pipeline & Workflow ManagementDevelop orchestration workflows (e.g., Airflow, Prefect, Azure Data Factory)Monitor and troubleshoot pipelines to ensure high availability and performanceImplement robust error handling, logging, and alertingData Quality & GovernanceEnsure data quality ...

technical specifications, and client-ready deliverables.Desirable Experience with advanced analytical techniques, such as machine learning, predictive modelling, or optimisation.Familiarity with data orchestration tools (e.g., Airflow, dbt, Prefect, Cloud Composer).Experience with containerisation or infrastructure-as-code (e.g., Docker, Kubernetes, Terraform).Knowledge of data warehousing concepts and technologies.Experience contributing ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark.Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.Implement continuous integration and continuous deployment ...

/CD, DevOps, and Infrastructure as Code practices for data platforms. Metadata management, lineage tooling, and data observability/monitoring experience. Familiarity with dbt, Apache Airflow, Azure Data Factory, Kafka, Event Hubs, or comparable orchestration/integration tools. Familiarity with enterprise AI platforms — Azure AI, AWS SageMaker, Google ...

engineers should be adept at Extract, Transform, Load (ETL) processes. This includes using frameworks such as the Medallion Architecture, as well as tools like Apache Airflow, Fivetran, Azure Data Factory, and Amazon Glue to automate data workflows and ensure data quality. Familiarity with cloud computing services, such … Cloud Platform, is increasingly important. Data engineers often leverage these platforms for scalable data storage and processing capabilities. Knowledge of big data frameworks like Apache Spark, Beam is crucial for handling large datasets and real-time data processing. Understanding data modelling techniques helps data engineers design efficient data structures ...

pipelinesCommercial experience working with Azure or AWS cloud platformsExperience with modern data platforms such as Databricks, Snowflake or RedshiftExperience using orchestration tools such as Airflow, Azure Data Factory or PrefectStrong understanding of data warehousing and data modelling principlesExcellent problem-solving skills with a delivery-focused mindsetComfortable working within cross ...

frameworks, orchestration tooling, and data engineering best practices. Familiarity with technologies such as Spark, Python, SQL, Delta Lake, Unity Catalog, dbt, Azure Data Factory, Airflow, Kafka, or equivalent tooling. Relevant certifications such as Snowflake SnowPro, Databricks Certified Data Engineer, or cloud platform certifications would be an advantage. Skills: Strong ...

analytics libraries (e.g., pandas, numpy, matplotlib)Exposure to DevOps, DataOps, and MLOps concepts and best practicesExperience with core technologies such as Python, PySpark, SQL, Airflow, Databricks, Kedro, Dask/RAPIDS, Docker, Kubernetes, and cloud services (AWS/GCP/Azure)Experience with Generative AI (GenAI) or agentic systems ...

MLOps, will be advantageousWhile we advocate using the right tech for the right task, we often leverage: Python, PySpark, the PyData stack, SQL, Airflow, Databricks, Kedro (our open-source data pipelining framework), Dask/RAPIDS, Docker, Kubernetes, and cloud solutions such as AWS, GCP, and AzureExperience with Generative ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow.Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time streams … preferred), including data modelling and design skills.Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data.Proven experience with Apache Airflow for data orchestration and scheduling at scale.Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must.Definition and design ...

analytics infrastructure. Set engineering standards, tooling decisions, and technical direction across the data stack (Python, SQL, DBT, AWS: S3, Glue, EMR, Lambda, Aurora, Airflow, Terraform). Ensure data availability, integrity, and performance as the platform scales globally. Drive best practices in data quality, automated testing, anomaly detection ...

Terraform and Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery.Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines.Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command ...

automated testing and data quality practice. A collaborative mindset and a genuine interest in building things well. Desirable Experience with orchestration tools such as Airflow, Dagster, Prefect or Azure Data Factory. Experience working within AWS, Azure, GCP or another enterprise cloud ecosystem. Experience with complex customer, transactional, digital ...