1 to 25 of 55 Apache Airflow Jobs in England

LGBTQ+ business community. Please do not contact the recruiter directly. About the role: As a Senior Support Engineer for our Data Platform, Snowflake and Apache Airflow, you will be a key part of the St. James's Place(SJP) infrastructure team You will be responsible for ensuring … services we deliver. What you'll be doing: Provide expert-level support to customers using the SJP Data Platform technologies including Snowflake and Apache Airflow. Supporting incident resolution for the platform and proactively monitoring and maintaining it to ensure high level of operational resilience. Appling SJP IT policies ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business … Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow or similar tools Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems Preferred qualifications, capabilities, and skills Data modeling ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business … Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow or similar tools Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems Preferred qualifications, capabilities, and skills Data modeling ...

Senior Data Engineer - Python, SQL, Airflow, Azure, Pyspark, Kubernetes Salary: Up to £80,000 per annum Benefits: Excellent benefits package including annual bonus, free onsite gym, and company pension scheme. About the Role We are seeking an experienced Senior Data Engineer to join an innovative SaaS business that … Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

pipelines and warehouses. Strong SQL skills and proficiency in Python or another relevant programming language. Experience with modern data engineering tools, such as dbt, Airflow or Spark. Experience with cloud data warehouses, such as Snowflake, BigQuery or Amazon Redshift. A solid understanding of data modelling, ETL/ ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working ...

Python and SQL programming skills, with experience developing and maintaining data pipelines Familiarity with ETL/ELT technologies and data streaming tools such as Airflow, Azure, SSIS, Spark or Kafka Experience working with databases and data platforms such as PostgreSQL, SQL Server, Oracle, Snowflake or similar technologies Knowledge ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

operational issues. Understanding of platform security, resilience, scalability and availability. Experience working within Agile and/or DevOps delivery environments . Desirable: Helm, ArgoCD, Airflow, dbt, Bash, Ansible, Packer, Elasticsearch/OpenSearch, Prometheus, Grafana, ELK or Datadog. Experience working within public sector, utilities or other regulated environments is advantageous. ...

compliant solutions that align with GDPR and healthcare data governance standards Stack: Snowflake | Snowflake Cortex | AWS - S3, Glue, Lambda, Step Functions | Python | SQL | dbt | Airflow| Terraform | Docker | GitHub CI/CD Requirements Must have advanced SQL and Python + experience delivering modern cloud-based data platforms Must have enterprise ...

running production pipelines end to end Excellent Python and SQL Solid grasp of data modelling, warehousing and lakehouse concepts Hands-on with orchestration (e.g. Airflow, Dagster) and distributed processing (e.g. Spark) Experience with cloud data platforms (AWS, Azure or GCP) and containerised deployment (Docker, Kubernetes) Good engineering practice: version ...

pipelines Strong programming skills in Python and SQL or similar Experience with technologies such as Spark, Databricks, AWS, Azure, GCP and/or Airflow Strong focus on data quality, testing and reliability Solid software engineering fundamentals and high engineering standards Strong communication and collaborative skills Comfortable working with ambiguity ...

analyse large datasets and communicate insights effectively. Experience with LLMs, generative AI, and prompt engineering. Knowledge of MLOps tools and frameworks (MLflow, Kubeflow, Airflow). Familiarity with SQL, NoSQL, and distributed data technologies. Experience deploying AI models in production environments. Publications, patents, or open-source contributions in ML/ ...

analyse large datasets and communicate insights effectively. Experience with LLMs, generative AI, and prompt engineering. Knowledge of MLOps tools and frameworks (MLflow, Kubeflow, Airflow). Familiarity with SQL, NoSQL, and distributed data technologies. Experience deploying AI models in production environments. Publications, patents, or open-source contributions in ML/ ...

relational databases (e.g., PostgreSQL, DuckDB). Experience with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity ...

hands-on experience in at least one data area and one platform area below. Data experience Orchestration: operating pipelines with tools such as Airflow, Dagster or Prefect, including dependencies, retries, backfills, reruns or schema changes. Storage and modelling: designing schemas and storage for relational databases, warehouses or lakehouses ...

lake, lakehouse, warehouse or large-scale analytical platform delivery. • Python, Bash or other scripting and automation approaches. • Workflow orchestration and scheduling tools, such as Airflow, TWS, JS7 or similar. • CI/CD, version control and modern engineering delivery practices. • Data quality, testing, observability and operational resilience. • Working with enterprise ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise • Observability ...

solution design responsibility. You'll ideally have: Strong commercial experience with Python & Snowflake Experience mentoring or technically leading other engineers Experience with dbt, Airflow or similar data engineering tooling A good understanding of data modelling and modern data platform architecture Experience working with AWS (ideally) or another major cloud ...

models, schemas to support business requirements Develop and maintain data ingestion and processing systems using various tools andtechnologies, such as SQL, NoSQL, ETL, Luigi, Airflow, Argo, etc. Implement data storage solutions using different types of databases, such asrelational, non-relational, or cloud-based. Working collaboratively with the client …/Azure SQL, PostgreSQL) You have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving ...