4 of 4 Apache Airflow Jobs in Milton Keynes

automation to ensure successful project delivery, adhering to client timelines and quality standards. Implement and manage real-time and batch data processing frameworks (e.g., Apache Kafka, Apache Spark, Google Cloud Dataproc) in line with project needs. Build and maintain robust monitoring, logging, and alerting systems for client projects … languages like Python, Bash, or Go to automate tasks and build necessary tools. Expertise in designing and optimising data pipelines using frameworks like Apache Airflow or equivalent. Demonstrated experience with real-time and batch data processing frameworks, including Apache Kafka, Apache Spark, or Google Cloud Dataflow. ...

maintain highly reliable and well-tested data and ML pipelines to extract, transform, and structure large datasets for ML applications. Expertise in using Apache Airflow (or similar orchestration tools like Prefect/Dagster) to define, schedule, and monitor complex data and ML workflows (DAGs). Implement comprehensive software … engineering teams. Bonus Skills to Stand Out (Optional): Experience with CI/CD tooling (e.g., GitHub Actions, Jenkins) specifically for ML pipelines and Airflow DAG deployment. Experience with data quality monitoring tools and frameworks. Master's or PhD in Computer Science, Data Science, or a related field. Relevant certifications ...

developing, evaluating and orchestrating agentic workflows MLOps and Distributed Systems: Hands-on experience with distributed systems and MLOps tooling such as Kubernetes, Docker, MLflow, Airflow, Terraform, and CI/CD. Preferred Skills Data Streaming and Orchestration: Familiarity with tools such as Kafka, Flink, Spark, dbt, and Airflow. Multi-Modal ...

scale. Deep understanding of distributed systems design and implementation. Strong Kubernetes experience managing production workloads. Proficiency with workflow orchestration frameworks such as Temporal, Airflow, or similar. Hands-on experience with AWS (Azure familiarity is a plus). Exposure to large-scale data frameworks such as Spark, Athena, or equivalents. ...