10 of 10 Apache Spark Jobs in the North West

will initially be for 6 months. It is also fully remote. Key Responsibilities • Design, develop, and maintain batch and streaming data pipelines using Databricks (Apache Spark) • Build and optimize ETL/ELT workflows for large-scale structured and unstructured data • Implement Delta Lake architectures (Bronze/Silver/… Gold layers) • Integrate data from multiple sources (databases, APIs, event streams, files) • Optimize Spark jobs for performance, scalability, and cost • Manage data quality, validation, and monitoring • Collaborate with analytics and ML teams to support reporting and model development • Implement CI/CD, version control, and automated testing for data ...

Charity Volunteer Days Cycle to work scheme And More.. DATA ENGINEER – ESSTENTIAL SKILLS Proven experience building data pipelines using Databricks . Strong understanding of Apache Spark (PySpark or Scala) and Structured Streaming . Experience working with Kafka (MSK) and handling real-time data . Good knowledge of Delta ...

sectors. Strong knowledge of data fabric architectures, metadata management, and data governance principles. Experience with SQL, ETL tools, and big data technologies (e.g., Hadoop, Spark, Kafka). Familiarity with cloud platforms (AWS, Azure, or Google Cloud) and data integration frameworks. Strong problem-solving skills and ability to work ...

organisation Supporting a move towards a product-aligned, hub-based operating model Technology Stack Cloud-first environment with Microsoft Azure Fabric Python, PyTorch SQL Apache Spark Azure Data Factory (ADF) Git What We’re Looking For 3–4 years’ experience in AI/ML engineering, data engineering ...

organisation Supporting a move towards a product-aligned, hub-based operating model Technology Stack Cloud-first environment with Microsoft Azure Fabric Python, PyTorch SQL Apache Spark Azure Data Factory (ADF) Git What We're Looking For 3-4 years' experience in AI/ML engineering, data engineering ...

containerised workloads (Docker, Kubernetes, Kubeflow, etc.) and integrating with CI/CD tools (Jenkins, GitHub Actions, GitLab CI). Familiarity with distributed computing frameworks (Spark, Ray, TensorFlow Distributed, PyTorch Distributed). Strong understanding of monitoring, logging, and observability for large-scale ML systems. Experience in cost optimisation for compute ...

DevOps, with the freedom to shape how GenAI and ML are used across the platform. Tech stack: Python PyTorch or TensorFlow scikit-learn Spark Kafka Databricks LangChain and RAG-based GenAI systems Cloud-native Azure environment Important Details: Fully remote across ...

will be able to demonstrate skills and experience in many of the following: - Expertise in designing production-grade data pipelines using Python, Scala, Spark, and SQL - Deep knowledge of AWS Cloud Platforms (EMR, Glue, Redshift, Kinesis, Lambda, DynamoDB) - Experience with data processing across structured and unstructured sources - Strong scripting ...

Deep expertise in search and recommendation systems (e.g., semantic embeddings, learning-to-rank, personalisation algorithms). Hands-on experience with modern ML toolchains — Python, Spark, and frameworks such as PyTorch or TensorFlow. Strong grounding in experimental design, A/B testing, and the use of offline/online metrics ...

kernels Experience with RLHF pipelines (PPO, DPO, ORPO) Experience training or deploying multimodal or diffusion models Experience in large-scale data processing (Apache Arrow, Spark, Ray) Prior work in a research lab (Google Brain, DeepMind, FAIR, Anthropic, OpenAI ...