1 to 25 of 241 Apache Airflow Jobs in London

Senior Astronomer Data Engineer

Location
City Of London, England, United Kingdom
Consultant to support the implementation and deployment of Astronomer (Astro) within a leading asset management organisation. Job Responsibilities/Objectives Astronomer is built around Apache Airflow and is used to develop, schedule, automate and manage complex data pipelines. The organisation is currently implementing the platform and requires … ownership of the implementation and providing training and support to upskill the wider team. Experience with Astronomer/Astro is essential, alongside knowledge of Apache Airflow, dbt and Snowflake. Key Responsibilities Lead the implementation, build and deployment of Astronomer (Astro). Configure and manage Apache Airflow ...

Solution Architect

Location
Greater London, England, United Kingdom
integration, and production support. Good understanding of ETL/ELT, data warehousing, data lakes, dimensional modelling, batch processing, and stream processing. Experience with Apache Beam, Kafka, REST APIs, Docker, Kubernetes, or GKE. Experience with data formats such as JSON, CSV, Avro, and Parquet. Working knowledge of Git, automated testing … observability, and release management. Ability to design solutions for scalability, resilience, security, performance, operability, and cost efficiency. Preferred Skills Experience with Cloud Composer or Apache Airflow, Dataproc or Spark, Dataform or dbt, and metadata-driven pipeline frameworks. Exposure to Terraform or another infrastructure-as-code tool. Knowledge ...

Sales Engineer, EMEA

Location
City Of London, England, United Kingdom
bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than … this role: Astronomer is on a mission to make DataOps a first-class discipline in every modern data organization. As the driving force behind Apache Airflow, we're powering mission-critical pipelines at scale for companies like Condé Nast, Rappi, Sonos, and more. As a Sales Engineer ...

Staff Data Engineer | Contract | U ...

Location
Greater London, England, United Kingdom
practices. Responsibilities will include: Providing Staff-level technical leadership across complex data engineering and modernisation initiatives. Remaining deeply hands‐on with Python, PySpark and Apache Spark , independently building and proving engineering solutions. Designing and optimising scalable data pipelines and processing frameworks across AWS . Tackling complex Spark performance challenges … including shuffles, joins, data skew, partitioning and memory optimisation. Designing and challenging engineering approaches across AWS Glue, Apache Airflow and associated technologies. Reviewing existing code, identifying performance bottlenecks and driving improvements in engineering quality. Establishing reusable engineering patterns, frameworks and approaches to accelerate large-scale data migrations. Applying ...

Machine Learning Engineer - Python & LLM Systems

Location
Greater London, England, United Kingdom
Build repeatable evaluations, investigate failure cases and use the results to guide model and pipeline improvements. Orchestrate data and model workflows. Build and maintain Apache Airflow pipelines, with monitoring, retries and clear approaches to failure recovery. Deploy and operate AI services. Work with AWS services including Lambda … through unfamiliar problems. You’re comfortable discussing trade-offs, asking questions and taking responsibility for delivery. Useful additional experience LLMs or NLP, GPU workloads, Apache Airflow, AWS, infrastructure-as-code tools such as Pulumi, or financial-market data. You don’t need experience with every tool ...

Data Engineer

Location
Greater London, England, United Kingdom
Knowledge of: MLOps Feature Engineering Exposure to data analysis and ML tools such as: scikit-learn pandas NumPy XGBoost LightGBM Familiarity with orchestration tools: Apache Oozie Apache Airflow Why you’ll love it here We want to make Prima a happy and empowering place to work. ...

Senior Data Engineer

Location
Greater London, England, United Kingdom
Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

Principal Data Engineer

Location
Greater London, England, United Kingdom
Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

Principal Data Engineer

Location
Greater London, England, United Kingdom
stores). Hands-on experience building, scaling, and shipping large-scale data platforms in production. Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop). Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis). End-to-end exposure … data pipeline lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster). Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg). Advanced SQL skills and proficiency in Python. Strong background ...

Data Technical Lead

Location
Greater London, England, United Kingdom
platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation … quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well-governed ...

Data Engineer III

Location
Greater London, England, United Kingdom
experience designing and building scalable data pipelines (ETL/ELT) for batch and streaming use cases using modern technologies (e.g., Python, SQL, Spark, Flink, Airflow) and advanced knowledge on the workings of these. Hands-on experience in building and operating workloads Databricks. Hands-on experience creating data visualizations ...

Senior Data Engineer

Location
Greater London, England, United Kingdom
Athena, Lambda, S3, Glue and Step Functions, but we're open to transferable experience solving similar problems. Experience with Snowflake, Spark, Scala or Airflow, or transferable experience with comparable technologies and solving similar data engineering challenges. Experience with Git, GitHub and CI/CD practices, including tools such ...

Data Engineer

Location
Greater London, England, United Kingdom
quality, testing, monitoring, and observability solutions, with tools such as Monte Carlo . Experience with workflow orchestration tools, preferably Prefect , or alternatives such as Airflow, Dagster, or Azure Data Factory. Understanding of event‐driven architectures, distributed data processing, and large‐scale data platforms. Experience with source control ...

Data Engineer (Newcastle upon Tyne)

Hiring Organisation
Catalyst
Location
City of London, South East, United Kingdom
Employment Type
Permanent Contract, On-site
multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

Senior Data Platform Engineer — Real-Time & Cloud-Native

Location
Greater London, England, United Kingdom
optimise batch and real-time pipelines Work with streaming and event-driven tech such as Kafka, Flink, Kinesis, Pub/Sub Orchestrate workflows using Airflow, Dataflow, Glue Work at Scale Process and transform large datasets using Spark and Flink Design systems that perform in production - not just on paper ...

Data Engineer

Location
Greater London, England, United Kingdom
analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases, and can integrate the relevant data quality checks ...

Senior Data Plaform Engineer

Hiring Organisation
Aker Systems Limited
Location
Central London, South East, United Kingdom
Employment Type
Permanent Contract, On-site
operational issues. Understanding of platform security, resilience, scalability and availability. Experience working within Agile and/or DevOps delivery environments . Desirable: Helm, ArgoCD, Airflow, dbt, Bash, Ansible, Packer, Elasticsearch/OpenSearch, Prometheus, Grafana, ELK or Datadog. Experience working within public sector, utilities or other regulated environments is advantageous. ...

Data Engineer

Location
Greater London, England, United Kingdom
translate them into a practical, maintainable data solution Strong problem‐solving and analytical skills Orchestration and monitoring tools, e.g. Databricks Workflows, dbt or Airflow (desirable) Integrating SaaS platforms through APIs, e.g. Salesforce or Workday (desirable) Building data catalogues, dictionaries or certified datasets, e.g. Unity Catalog, Microsoft Purview or Power ...

Data Engineer

Location
Greater London, England, United Kingdom
adhere to the D.R.Y principle, minimising manual intervention and promote automation. We're looking for these essential skills Hands‐on experience with DBT and Apache Airflow, building scalable data transformations and DAGs to orchestrate complex, reliable data pipelines. Manage task dependencies, automated retries and error handling to ensure ...

Lead Data Engineer

Location
Greater London, England, United Kingdom
Athena, Lambda, S3, Glue and Step Functions, but we're open to transferable experience solving similar problems. Experience with Snowflake, Spark, Scala or Airflow, or transferable experience with comparable technologies and solving similar data engineering challenges. Experience with Git, GitHub and CI/CD practices, including tools such ...

Data Engineer

Location
Greater London, England, United Kingdom
/CD, version control, and testing practices Strong problem-solving and communication skills Desirable Skills: Experience with tools such as Databricks, Data Factory, Fabric, Airflow, DBT, or equivalent Experience working with structured and semi-structured data sources Understanding of data security, governance, and compliance requirements Our culture and benefits ...

Senior Martech Data Engineer

Location
Greater London, England, United Kingdom
warehousing, data pipelines and data transformation (extract, transform, and load) processes and supporting technologies such as Google Dataflow, DBT, EMR, CI/CD Pipelines, Airflow DAGs, and other analytics tools. Experience with cloud-based data infrastructures (Ideally GCP, but AWS or Azure would also suffice). Expertise in managing ...

AI Engineer

Location
Greater London, England, United Kingdom
Spark, Databricks, Hadoop or cloud-native alternatives).* Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow).* Strong experience with data engineering concepts (ETL/ELT, data integration, warehousing, vector databases/search)**A little bit more about us... ...

Senior Data Engineer London

Location
Greater London, England, United Kingdom
software engineering experience using Python and/or other modern programming languages. Experience developing and orchestrating data pipelines using technologies such as DBT, Apache Spark, Airflow, Dagster, Kafka, Flink, or equivalent tooling. Solid understanding of data lakehouse, warehouse, and medallion architecture patterns. Experience applying DevOps and DataOps best ...

Data Engineer

Location
Greater London, England, United Kingdom
adhere to the D.R.Y principle, minimising manual intervention and promote automation. **We're looking for these essential skills** * Hands-on experience with DBT and Apache Airflow, building scalable data transformations and DAGs to orchestrate complex, reliable data pipelines.* Manage task dependencies, automated retries and error handling to ensure ...