11 of 11 Permanent Apache Spark Jobs in Yorkshire

will initially be for 6 months. It is also fully remote. Key Responsibilities • Design, develop, and maintain batch and streaming data pipelines using Databricks (Apache Spark) • Build and optimize ETL/ELT workflows for large-scale structured and unstructured data • Implement Delta Lake architectures (Bronze/Silver/… Gold layers) • Integrate data from multiple sources (databases, APIs, event streams, files) • Optimize Spark jobs for performance, scalability, and cost • Manage data quality, validation, and monitoring • Collaborate with analytics and ML teams to support reporting and model development • Implement CI/CD, version control, and automated testing for data ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

Experience building and deploying cloud data platforms Infrastructure-as-Code and CI/CD practices (GitHub, CodeBuild, pipelines) Strong coding experience in Python (Pandas), Apache Spark, and/or Apache Iceberg Expertise in SQL and advanced data manipulation and analysis Experience with data warehouse design and data ...

platforms such as AWS, Azure, or GCP. Exposure to modern data tools such as Snowflake, Databricks, or BigQuery. Familiarity with streaming technologies (e.g., Kafka, Spark Streaming, Flink) is an advantage. Experience with orchestration and infrastructure tools such as Airflow, dbt, Prefect, CI/CD pipelines, and Terraform. What ...

containerised workloads (Docker, Kubernetes, Kubeflow, etc.) and integrating with CI/CD tools (Jenkins, GitHub Actions, GitLab CI). Familiarity with distributed computing frameworks (Spark, Ray, TensorFlow Distributed, PyTorch Distributed). Strong understanding of monitoring, logging, and observability for large-scale ML systems. Experience in cost optimisation for compute ...

DevOps, with the freedom to shape how GenAI and ML are used across the platform. Tech stack: Python PyTorch or TensorFlow scikit-learn Spark Kafka Databricks LangChain and RAG-based GenAI systems Cloud-native Azure environment Important Details: Fully remote across ...

Deep expertise in search and recommendation systems (e.g., semantic embeddings, learning-to-rank, personalisation algorithms). Hands-on experience with modern ML toolchains — Python, Spark, and frameworks such as PyTorch or TensorFlow. Strong grounding in experimental design, A/B testing, and the use of offline/online metrics ...

kernels Experience with RLHF pipelines (PPO, DPO, ORPO) Experience training or deploying multimodal or diffusion models Experience in large-scale data processing (Apache Arrow, Spark, Ray) Prior work in a research lab (Google Brain, DeepMind, FAIR, Anthropic, OpenAI ...

kernels Experience with RLHF pipelines (PPO, DPO, ORPO) Experience training or deploying multimodal or diffusion models Experience in large-scale data processing (Apache Arrow, Spark, Ray) Prior work in a research lab (Google Brain, DeepMind, FAIR, Anthropic, OpenAI ...

kernels Experience with RLHF pipelines (PPO, DPO, ORPO) Experience training or deploying multimodal or diffusion models Experience in large-scale data processing (Apache Arrow, Spark, Ray) Prior work in a research lab (Google Brain, DeepMind, FAIR, Anthropic, OpenAI ...

kernels Experience with RLHF pipelines (PPO, DPO, ORPO) Experience training or deploying multimodal or diffusion models Experience in large-scale data processing (Apache Arrow, Spark, Ray) Prior work in a research lab (Google Brain, DeepMind, FAIR, Anthropic, OpenAI ...