26 to 50 of 86 Apache Airflow Jobs in the UK

pipelines end to end. Modern data stack Experience with modern ELT, transformation and orchestration technologies. Our environment includes technologies such as dbt, Dagster/Airflow and Snowflake, although experience with comparable technologies is equally relevant. Cloud infrastructure Experience designing and implementing secure, scalable solutions on a major cloud platform. ...

data pipelines, ELT workflows, and contemporary cloud data architectures. Experience with current cloud data platforms such as Snowflake, Azure Data Lake, Azure SQL, dbt, Airflow, or similar tools. Ability to understand and work with SQL queries and data models. Knowledge of data governance, privacy, security, regulatory reporting, and responsible ...

with reverse ETL and moving data between warehouses and operational systems Strong understanding of APIs Familiarity with modern data engineering tools such as dbt, Airflow, Fivetran or Census Excellent stakeholder management skills If you're an experienced Senior Data Engineer who enjoys solving complex integration challenges, working with modern ...

skills Experience building data pipelines and integrating data from multiple sources Exposure to technologies such as Power BI, Snowflake, SQL Server, PostgreSQL, Spark, Kafka, Airflow, Docker, and Kubernetes Experience within engineering, manufacturing, industrial, or other complex regulated environments would be advantageous Excellent stakeholder management and problem-solving abilities What ...

pipelines Strong programming skills in Python and SQL or similar Experience with technologies such as Spark, Databricks, AWS, Azure, GCP and/or Airflow Strong focus on data quality, testing and reliability Solid software engineering fundamentals and high engineering standards Strong communication and collaborative skills Comfortable working with ambiguity ...

ADLS, and Azure Functions. Proven track record of designing and implementing metadata-driven data pipelines. Deep expertise in orchestration and data workflow automation e.g. Airflow, DBT. Strong understanding of CI/CD practices for data engineering. Experience with infrastructure as code (Terraform, Bicep, or ARM templates) (preferred). Solid ...

hands-on experience in at least one data area and one platform area below. Data experience Orchestration: operating pipelines with tools such as Airflow, Dagster or Prefect, including dependencies, retries, backfills, reruns or schema changes. Storage and modelling: designing schemas and storage for relational databases, warehouses or lakehouses ...

novelty. You don't need to know all of it on day one. Data and compute: S3, Athena, Spark on EMR, SQLOrchestration and MLOps: Airflow, MLflow, Docker, ECS, Terraform, GitHub Actions for CI/CDModels: foundation models served through Amazon BedrockAgents: LangGraph for orchestration, deployed and run on Amazon ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

Snowpark Iceberg Tables Secure Data Sharing Snowflake Cortex Data Integration Fivetran SnapLogic Enterprise SaaS integration patterns Data Transformation & Orchestration dbt Cloud/dbt Core Apache Airflow (MWAA) CI/CD pipelines Event-driven orchestration Data Governance & Observability Collibra Data Catalogue Data Lineage Data Governance Frameworks Acceldata Data Observability ...

need both skillsets below - strength in one is enough: Hands-on: building AI/ML pipelines or prototypes; vector databases; prompt engineering; orchestration tools (Airflow/Prefect); Git Advisory: designing AI/data architectures; Agentic AI principles (LangGraph, CrewAI, AutoGen); mentoring specialists without formal authority; AI governance (security, privacy ...

transformation. Pipeline & Analytics Tooling: Hands-on experience with modern data stacks such as Databricks , Azure Data Factory/Synapse , AWS Glue/Redshift , or Airflow/Stream Analytics . Engineering Best Practices: Proven track record applying software craftsmanship principles to data pipelines, including automated CI/CD ( GitHub Actions ...

using Python and modern data frameworks. Integrate data from sensors, databases, cloud platforms, and third-party services. Orchestrate data workflows using tools such as Airflow and Prefect. Create IoT and Real-Time Data SolutionsCollect, normalize, and process telemetry from connected devices. Implement edge-to-cloud data flows using MQTT ...

issuesData Engineers must have:2:1 or above Hons. degree or equivalent in mathematics, computer science or comparable scientific discipline. PythonBuild ETL Pipelines (e.g., Airflow)JavascriptBuild and CI/CD toolsWrite SQL codeCreate and implementing Rest APIsGIT and GIT branchingBI Tools such as Looker, Qlik, Power BI, TableauContainerisation technologiesGoogle ...

orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic and transparency of underlying code Owned a project across a whole ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise • Observability ...

solution design responsibility. You'll ideally have: Strong commercial experience with Python & Snowflake Experience mentoring or technically leading other engineers Experience with dbt, Airflow or similar data engineering tooling A good understanding of data modelling and modern data platform architecture Experience working with AWS (ideally) or another major cloud ...

technical leadership and solution design responsibility. You'll ideally have: Strong commercial experience with Python & SnowflakeExperience mentoring or technically leading other engineersExperience with dbt, Airflow or similar data engineering toolingA good understanding of data modelling and modern data platform architectureExperience working with AWS (ideally) or another major cloud platformThe ...

pipelines. Proficiency in SQL, Python, AWS, and other modern data engineering technologies. Experience with version control tools (e.g., Git) and workflow orchestration platforms (e.g., Airflow). Proven ability to test, troubleshoot, and improve data platforms, ensuring high levels of reliability and data quality. Strong interpersonal and communication skills with ...

models, schemas to support business requirements Develop and maintain data ingestion and processing systems using various tools andtechnologies, such as SQL, NoSQL, ETL, Luigi, Airflow, Argo, etc. Implement data storage solutions using different types of databases, such asrelational, non-relational, or cloud-based. Working collaboratively with the client …/Azure SQL, PostgreSQL) You have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving ...

knowledge of AI infrastructure. Good proficiency in programming languages such as Python, Java, or C++. Experience with data pipeline and workflow management tools (e.g., Apache Airflow, Kubeflow). Strong problem-solving skills and ability to work in a fast-paced environment. Excellent communication and collaboration skills. Significant experience ...

necessary infrastructure, resources, and interfaces to enable data loading and LLM workflows. Use Python/Java and large-scale data workflow orchestration platforms e.g. Airflow to construct software artifacts for ETL, interfacing with diverse data formats and storage technologies, and incorporate them into robust data workflows and dynamic systems ...

external CLI tools/subprocesses safely. You might also have experience of the following: Direct experience with a workflow orchestration framework (Prefect, Airflow, Dagster, or similar).Experience with containerization (Docker, or HPC-oriented tools like Apptainer/Singularity).Exposure to systems that coordinate with external schedulers/queues ...

bandwidth. The goal: no software rewrites, predictable time to production, early integration, and fast iteration. You'll work with tools such as DBT, Python, Apache Airflow, Spark, Iceberg; AWS services like Lambda, Bedrock, and SageMaker, containers, and whatever adds the most value. You'll design tools that ...

Nice to have Consulting or client-facing delivery experience Experience evaluating and monitoring LLM systems (testing, guardrails, cost control) MLOps tools such as MLflow, Airflow or CI/CD pipelines We Are Dcoded is acting as an employment agency in relation to this permanent vacancy. WKCL1_UKTJ ...