1 to 25 of 39 Apache Airflow Jobs in London

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … through to validated delivery Architect and build the Claude Code agentic workflow: Parse existing data movement pipelines and logic Generate equivalent PySpark transformations Produce Airflow DAGs Operate, iterate, and scale across the full data pipeline estate Ensure rigorous creation and management of the PySpark data processing layer within ...

systems (CMS), subscription platforms, ad tech, and social media. Proven ability to automate and optimise data workflows , using modern ETL/ELT tools (e.g., Airflow, dbt, Apache Spark) to ensure timely and reliable delivery of data. Experience building robust data models and reporting layers to support performance dashboards … hands-on experience in a data engineering role. Tools & Technologies : Databases : Proficient in SQL databases. Workflow Management Tools : Experience with orchestration platforms such as Apache Airflow . Programming Languages : Skilled in one or more of the following languages, i.e.: Python , Java , Go. Cloud Infrastructure : Strong understanding of cloud ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Notebooks for data exploration and development Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ETL/ELT development and orchestration tools (e.g., Apache Airflow, dbt, Prefect). Knowledge of data modelling, data warehousing, and lakehouse architectures. Familiarity with DevOps practices, CI/CD pipelines, and infrastructure ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

processin g.Cloud Pub/Su b: For real-time messaging and event ingestio n.Cloud Storag e: For scalable and durable object storag e.Cloud Composer (Apache Airflow ): For workflow orchestration and pipeline managemen t.Cloud Datapro c: For managed Hadoop and Spark clusters (good to have ).Cloud SQL/Cloud ...

Responsibilities Pipeline Development: Design, develop, and maintain robust end-to-end data pipelines and orchestration workflows using Python , Apache Airflow , and Cloud Composer on GCP. Data Modernization: Bridge the gap between legacy/on-premise database environments and modern cloud architectures, ensuring seamless data ingestion and migration. Optimization … Experienc eGCP Core Stack : Proven production-grade experience building and managing cloud data solutions on Google Cloud Platfor m (specifically BigQuery, Cloud Composer/Airflow, Cloud Storag e, and Dataflo w) .Programming & Scripting : Strong hands-on coding capability in Pytho n for data engineering, service development, and pipeline automation ...

working in secure, regulated, or complex enterprise environments Exposure to large-scale data platforms or transformation programmes Knowledge of tools such as Spark, Databricks, Airflow, or similar Experience in consulting or client-facing delivery environments Security Requirements Must be a sole UK citizen Must be eligible for Security Clearance ...

Experience using Git in a team environment Hands-on experience using AI coding tools (e.g., Copilot, Cursor, Claude, Codex) Nice to have: Experience with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

1classification ) Strong SQL&Python Hands-onETL/ELTexperience(e.g.,Matillion,Talend,Fivetran,ADF) Experiencewith Snowflake (AWS/Azure/GCPbeneficial) Strongdatamodellingknowledge Understandingofdatagovernanceandsecuritybestpractices Abilitytotranslatebusinessneedsintoscalablesolutions Desirable Airflow,dbt,Kafka,CI/CD,BItools,governanceplatforms,financialservicesexperience. Interestedinbuildingmodern,enterprise-scaledataplatformsinahighlyregulatedenvironment Applynow ...

data pipelines, managing large datasets, and enabling engineering and AI teams to deliver high-performance data-driven products. Key Technical Requirements • Python • SQL • Spark • Airflow • Data pipelines • AWS or GCP Experience Required • 4+ years in data engineering • Experience working with large-scale structured and unstructured datasets • Strong ETL development ...

real-time processing. Build and maintain data ingestion frameworks from multiple data sources. Develop scalable data models and data warehouse solutions. Implement workflow orchestration (Airflow/Cloud Composer) including retries, dependencies, and monitoring. Ensure data quality, validation, and governance mechanisms. Design and manage streaming pipelines (Kafka/ ...

environments Solid experience with SQL and data modelling concepts Strong analytical and problem-solving skills Nice to Have Skills Experience with data pipeline tools (Airflow, Kafka, Spark) Knowledge of cloud platforms (AWS, Azure or GCP) Familiarity with relational and NoSQL databases Additional programming experience (Python, Scala) Essential Requirements Active ...

Nice to Have: * Microsoft Azure Solutions Architect Expert* AWS Solutions Architect Professional* Databricks Certifications* CDMP Certification* Terraform, Bicep or Infrastructure-as-Code experience* dbt, Airflow or modern orchestration frameworks* Experience supporting AI, GenAI or RAG-based architectures* Experience within Financial Services, Healthcare, Life Sciences, Energy, Retail or Manufacturing Robert ...

Experience with embeddings, vector databases, RAG, LLM-based enrichment, or knowledge graph completion. Experience with MLOps tools such as Docker, Kubernetes, MLflow, Weights & Biases, Airflow, or similar. Experience with AWS, GCP, or Azure. Experience in construction, architecture, BIM, building materials, technical product data, catalogues, standards, or compliance-heavy data ...

production-grade data pipelines Solid understanding of integration patterns including REST APIs, webhooks, OAuth and event-driven architectures Experience with orchestration tools such as Airflow, Prefect or Dagster Familiarity with cloud environments (AWS, GCP or Azure) Experience with containerisation tools such as Docker and Kubernetes Proven experience integrating multiple ...

orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic and transparency of underlying code Owned a project across a whole ...

Experience working within cloud environments (AWS or Azure) Strong SQL skills and experience working with large-scale datasets Familiarity with orchestration tools such as Airflow or similar Good understanding of data modelling and distributed data processing concepts Comfortable working across multiple client environments and stakeholders 🎁 What’s on Offer ...

preferred) or similar platforms: Snowflake, BigQuery, Redshift, Synapse, or equivalent Strong proficiency with: SQL (advanced level) Spark/PySpark Data pipeline orchestration tools (e.g., Airflow, ADF, etc.) Data Modelling Extensive experience designing: Fact and dimension tables Star and snowflake schemas Aggregation layers and semantic models Data Migration & Mapping Demonstrated ...

organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design, Data Governance ...

retrieval augmented generation (RAG) architectures. * Strong knowledge of cloud platforms including AWS or Azure. * Experience with data engineering and orchestration tools such as Databricks, Airflow, Docker, and Kubernetes is desirable. * Must have previous experience working within a small to medium-sized business (SME), start-up, or scale-up environment. ...

GitHub Copilot) with significant examples of delivery improvement AWS (S3, Lambda, EC2, Glue), Snowflake, Git, CI/CD, Datadog, MS SQL Server, REST, GraphQL, Airflow (as many as possible) Degree educated or higher in a relevant technical discipline from a leading university This is a £650-700/ ...

creating. You'll have experience with: AWS and cloud-native technologies Snowflake or similar cloud data platforms Data orchestration tools such as Airflow, Prefect or similar Amazon S3 and modern data architectures Postgres and semi-structured data models Event-driven architectures and scalable systems You'll understand how data ...