1 to 25 of 35 Permanent Apache Airflow Jobs in London

possible! Diverse training opportunities and social benefits (e.g. UK pension schema) Strong hands-on experience working with modern Big Data technologies such as Apache Spark, Trino, Apache Kafka, Apache Hadoop, Apache HBase, Apache Nifi, Apache Airflow, Opensearch Proficiency in cloud-native technologies such ...

execute. You have deep experience designing, building, and operating production data pipelines, including owning reliability, monitoring, and failure recovery for orchestration systems such as Apache Airflow (or equivalent). You are fluent in modern data and cloud ecosystems, with hands-on experience (or the ability to ramp quickly … days a week in office Lean principles and Agile development practices Continuous deployment across all microservices Big Data ecosystem and SQL Databases (Apache Airflow, Iceberg, Spark/Thrift, PostgreSQL, Azure Hyperscale) Python as a primary backend language High-level frameworks (Flask, SQLAlchemy, Alembic, Pytest, Socket.IO) Amazon Web Services ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration ...

certifications (Data Analytics Specialty or Solutions Architect Associate). p]:pt-0 [&>p]:mb-2 [&>p]:my-0'> Experience with Airflow for orchestration. p]:pt-0 [&>p]:mb-2 [&>p]:my-0'> Exposure to big data frameworks such as Spark, Hadoop, or Presto. ...

Building Extract Transform and Load (ETL) pipeline (streaming/batch) using MPP frameworks (Spark, Beam, or other) Experience in orchestrating complex pipelines using Airflow, Dagster or other Proficiency in one or more programming languages: Python, Java, GO or other Experience with AWS or GCP and their products ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

databases (Pinecone, Weaviate, FAISS, Milvus) Data & Engineering Excellence Lead the evolution of modern data platforms (Databricks, Snowflake, Synapse, BigQuery) Ensure robust data engineering practices (Airflow, dbt, Prefect) and governance (Collibra, Purview) Drive scalable lakehouse architectures (Delta Lake, Iceberg, Hudi) Cloud, Infrastructure & MLOps Champion multi-cloud engineering across Azure ...

real-time processing. Build and maintain data ingestion frameworks from multiple data sources. Develop scalable data models and data warehouse solutions. Implement workflow orchestration (Airflow/Cloud Composer) including retries, dependencies, and monitoring. Ensure data quality, validation, and governance mechanisms. Design and manage streaming pipelines (Kafka/ ...

Glue or EKS Hands on experience with modern analytics stacks such as Snowflake, dbt and Iceberg Familiarity with orchestration tools such as Dagster or Airflow Strong understanding of data modelling concepts and analytics best practice A software engineering mindset, including Git, testing and CI/CD workflows What They ...

production-grade data pipelines Solid understanding of integration patterns including REST APIs, webhooks, OAuth and event-driven architectures Experience with orchestration tools such as Airflow, Prefect or Dagster Familiarity with cloud environments (AWS, GCP or Azure) Experience with containerisation tools such as Docker and Kubernetes Proven experience integrating multiple ...

/DS libraries (scikit-learn, numpy, LightlGBM, Pandas, LangChain/LangGraph, TensorFlow, etc..) PySpark AWS cloud infrastructure: EMR, ECS, Athena, etc. MLOps: Terraform, Docker, Airflow, MLFlow More information: Enjoy fantastic perks like private healthcare & dental insurance, a generous work from abroad policy, 2-for-1 share purchase plans ...

Redux, Redux Hooks, and other modern state management libraries. Kotlin on Android and Swift on iOS. Data Analytics/Engineering: SQL, Python, Snowflake, dbt, Airflow & Looker. AI at Marshmallow: Sierra.ai for conversational AI, Python and LangChain for agentic AI solutions, SageMaker and MLflow for Machine Learning, Cursor and Claude ...

with Python-based development workflows and tooling Technology Stack: Azure Cloud Services C# (.NET 10) Python React.js and modern JavaScript (ES6+) Snowflake (Data Warehouse) Apache Airflow Terraform dbt SQL Server ...

with Python-based development workflows and tooling Technology Stack: Azure Cloud Services C# (.NET 10) Python React.js and modern JavaScript (ES6+) Snowflake (Data Warehouse) Apache Airflow Terraform dbt SQL Server ...

technical solutions meet business needs. Experience with data ingestion tools, like Fivetran. Advantageous Exposure to deploying applications with Kubernetes. Experience with Data Orchestrator tools (Airflow, Prefect, etc.). Experience with Data Observability tools (Montecarlo, Great Expectations, etc.). Experience with Data Catalog tools (Amundsen, OpenMetadata, etc.). Benefits Winning ...

orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic and transparency of underlying code Owned a project across a whole ...

. ML workflow orchestration and serving: Operate the systems researchers use to launch experiments, data jobs, and production endpoints — workflow engines (e.g., Kubeflow Pipelines, Airflow), GPU schedulers (e.g., Volcano, Slurm), experiment trackers (e.g., MLflow, Weights & Biases), and managed‐inference platforms (e.g., Modal, Triton) — and maintain a launcher ...

complete features, though it's not the primary focus of the role. Our Stack (100% Open Source) Backend: Golang, Python Infrastructure & Orchestration: Docker, Kubernetes, Airflow, Salt, Ansible, Chef Data: PostgreSQL, Redis, Kafka API: REST, GraphQL Frontend: React with TypeScript What you’ll do Own backend systems — design, build … networking, filesystem concepts Configuration management experience with Salt, Ansible, Chef, or similar tools Container and orchestration expertise — Docker, Kubernetes, or related technologies Workflow orchestration — Apache Airflow or similar pipeline frameworks Database design skills — schema design and query optimization in PostgreSQL or similar RDBMS Distributed systems knowledge — you think ...

Experience working within cloud environments (AWS or Azure) Strong SQL skills and experience working with large-scale datasets Familiarity with orchestration tools such as Airflow or similar Good understanding of data modelling and distributed data processing concepts Comfortable working across multiple client environments and stakeholders 🎁 What’s on Offer ...

Kubernetes and modern containerised deployment patterns Strong experience with CI/CD pipelines, infrastructure-as-code and workflow orchestration Experience with tools such as Airflow or similar platform and orchestration technologies Good understanding of model observability, data quality, feature pipelines, lineage and reproducibility Experience designing scalable infrastructure ...

world data volumes. Developed data warehousing solutions with hands‐on experience building ETL pipelines, data modelling, and orchestration workflows using modern tools (e.g., dbt, Airflow, Spark), demonstrating the ability to transform raw data into reliable, performant analytics infrastructure that supports business decision‐making at scale. Collaborated effectively across teams ...

data stack including a mix of structured and unstructured data sources, orchestration tooling, and distributed storage. RDS Postgres, MongoDB, AWS Athena, Parquet, AWS Glue, Airflow, Python, Docker, S3, GraphQL, REST. You won't know all of it, you'll be strong in your core area and curious about ...

libraries (scikit-learn, NumPy, LightGBM, Pandas, LangChain/LangGraph, TensorFlow, etc.) PySpark AWS cloud infrastructure: EMR, ECS, Athena, etc MLOps/DevOps: Terraform, Docker, Airflow, MLFlow Monitoring and evaluation platforms including New Relic and Braintrust More information Enjoy fantastic perks like private healthcare & dental insurance, a generous work from ...

APIs can be exposed to LLM-based agents. Experience with search relevance & recommendation systems — ranking algorithms, semantic search, personalisation. Hands-on with Snowflake, Databricks, Airflow, Kafka . Why Join Us Work on firm-wide platforms used by thousands of data scientists, ML engineers, and analysts across JPMC Shape ...

Python, and Ansible Familiarity with containers (Docker, Kubernetes, LXC) Experience with monitoring tools and SQL databases Exposure to big data systems (e.g., Spark, HDFS, Airflow) Summary A great opportunity for someone who enjoys solving complex infrastructure challenges, automating systems, and working closely with engineering teams. Spectrum IT Recruitment (South ...