17 of 17 Apache Airflow Jobs in Manchester

automation to ensure successful project delivery, adhering to client timelines and quality standards. Implement and manage real-time and batch data processing frameworks (e.g., Apache Kafka, Apache Spark, Google Cloud Dataproc) in line with project needs. Build and maintain robust monitoring, logging, and alerting systems for client projects … languages like Python, Bash, or Go to automate tasks and build necessary tools. Expertise in designing and optimising data pipelines using frameworks like Apache Airflow or equivalent. Demonstrated experience with real-time and batch data processing frameworks, including Apache Kafka, Apache Spark, or Google Cloud Dataflow. ...

experience designing and building modern data platforms in AWS. ETL/Orchestration Expertise: Expertise in ETL/ELT design and data orchestration, specifically with Apache Airflow. SQL Mastery: Strong SQL skills with significant experience in query tuning and performance optimisation. Programming Proficiency: Proficiency in Python and Bash (for data ...

similar language. Proficiency in database technologies (SQL, NoSQL, time-series databases) and data modelling. Strong understanding of data pipeline orchestration tools (e.g., Apache Airflow, Kubernetes). You thrive when working as part of a team Comfortable in a fast-paced environment Have excellent written and verbal English skills ...

similar language. Proficiency in database technologies (SQL, NoSQL, time-series databases) and data modelling. Strong understanding of data pipeline orchestration tools (e.g., Apache Airflow, Kubernetes). You thrive when working as part of a team Comfortable in a fast-paced environment Have excellent written and verbal English skills ...

using concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical ...

maintain highly reliable and well-tested data and ML pipelines to extract, transform, and structure large datasets for ML applications. Expertise in using Apache Airflow (or similar orchestration tools like Prefect/Dagster) to define, schedule, and monitor complex data and ML workflows (DAGs). Implement comprehensive software … engineering teams. Bonus Skills to Stand Out (Optional): Experience with CI/CD tooling (e.g., GitHub Actions, Jenkins) specifically for ML pipelines and Airflow DAG deployment. Experience with data quality monitoring tools and frameworks. Master's or PhD in Computer Science, Data Science, or a related field. Relevant certifications ...

strong ability to learn it quickly. Strong SQL skills; working familiarity with Python for scripting and DBT automation. Experience integrating DBT with orchestration tools (Airflow, Dagster, Prefect, etc.). Solid understanding of modern data engineering, ELT patterns, and version-controlled analytics development. Preferred Prior experience operationalizing ML workflows inside ...

strong ability to learn it quickly. Strong SQL skills; working familiarity with Python for scripting and DBT automation. Experience integrating DBT with orchestration tools (Airflow, Dagster, Prefect, etc.). Solid understanding of modern data engineering, ELT patterns, and version-controlled analytics development. Preferred Prior experience operationalizing ML workflows inside ...

Excellent problem-solving and communication skills, with a focus on delivering high-quality solutions. > Understanding/hands on experience in Orchestration solutions such as Airflow > Deep knowledge of key non-functional requirements such as availability, scalability, operability, and maintainability ...

data pipelines at scale, you'll thrive here. Tech stack: Micro-services Container Platforms (Kubernetes, CRC, Docker) Big Data Technologies (Apache.... Spark, Flink, HDFS, Airflow, Trino, Iceberg, Hive) Dependency injection frameworks (Spring) Observability (Loki/Grafana) Large scale data processing (Kafka) CI/CD Build tools ( Maven, Git, Jenkins ...

strong ability to learn it quickly. Strong SQL skills; working familiarity with Python for scripting and DBT automation. Experience integrating DBT with orchestration tools (Airflow, Dagster, Prefect, etc.). Solid understanding of modern data engineering, ELT patterns, and version-controlled analytics development. Preferred Prior experience operationalizing ML workflows inside ...

. Develop robust backend systems and APIs to support workflows. Build and maintain CI/CD pipelines, infrastructure-as-code, and container orchestration (Kubernetes, Airflow, Databricks Jobs). Ensure best practices in model tracking, versioning, performance monitoring, logging, and retraining workflows. Collaborate with cross-functional teams to align technical ...

Principal Data Engineer: Experience as a lead or principal data engineer. Prior experience designing data platform(s) within GCP, working hands on with; Airflow, Big Query, Data Flow, Data Fusion, and Data Stream. Deep understanding of Data Mesh/decentralised design and Data Lake/Warehouse solutions. Previously ...

management. Requirements for the Principal Data Engineer: Experience as a lead or principal data engineer. Prior experience designing data platform(s) within GCP, across Airflow, Big Query, Data Flow, Data Fusion, and Data Stream. Deep understanding of Data Mesh design and Data Lake/Warehouse solutions. Previously led teams ...

experienced team members and fostering a culture of knowledge sharing Communicating clearly and effectively with diverse technical and non-technical audiences Tech stack Python , Airflow , Terraform Azure (but open to other cloud experience) DBT (nice to have) Exposure to ML pipelines beneficial ...

Technical Standards and Engineering Effectiveness within development squads. • Technical Experience: Demonstrate expertise in using SQL (Spark, Dremio), Python, GitHub and data orchestration tools (Airflow, Oozie) for data wrangling, building data pipelines and developing analytical interfaces. • AI-Assisted Analytics: Bring experience and curiosity towards AI-assisted analytics and machine learning … Strong knowledge and experience of using business intelligence, ETL frameworks and visualisation tools such as Tableau.• Experience with Python, GitHub, data orchestration tools (Oozie, Airflow) for data wrangling, building data pipelines and building analytical/web interfaces.• Good understanding of the full data lifecycle and understanding of typical enterprise ...

Technical Standards and Engineering Effectiveness within development squads. • Technical Experience: Demonstrate expertise in using SQL (Spark, Dremio), Python, GitHub and data orchestration tools (Airflow, Oozie) for data wrangling, building data pipelines and developing analytical interfaces. • AI-Assisted Analytics: Bring experience and curiosity towards AI-assisted analytics and machine learning … Strong knowledge and experience of using business intelligence, ETL frameworks and visualisation tools such as Tableau.• Experience with Python, GitHub, data orchestration tools (Oozie, Airflow) for data wrangling, building data pipelines and building analytical/web interfaces.• Good understanding of the full data lifecycle and understanding of typical enterprise ...