76 to 100 of 245 Apache Spark Jobs in England

of a forward-thinking company where data is central to strategic decision-making. We’re looking for someone who brings hands-on experience in streaming data architectures, particularly with Apache Kafka and Confluent Cloud, and is eager to shape the future of scalable, real-time data pipelines. You’ll work closely with both the core Data Engineering team and … the Data Science function, bridging the gap between model development and production-grade data infrastructure. What You’ll Do: Design, build, and maintain real-time data streaming pipelines using Apache Kafka and Confluent Cloud. Architect and implement robust, scalable data ingestion frameworks for batch and streaming use cases. Collaborate with stakeholders to deliver high-quality, reliable datasets to live … experience in a Data Engineering or related role. Strong experience with streaming technologies such as Kafka, Kafka Streams, and/or Confluent Cloud (must-have). Solid knowledge of Apache Spark and Databricks. Proficiency in Python for data processing and automation. Familiarity with NoSQL technologies (e.g., MongoDB, Cassandra, or DynamoDB). Exposure to machine learning pipelines or close More ❯

We use a variety of 3rd party packages, including MLFlow, Seldon for ML model tracking and deployment, Kubernetes for hosting models, Argo and Git for CI/CD automation, Spark for big data processing. This is a rapidly changing field and we are deeply involved in open source community to help shape the technology evolution and are constantly looking … for components to adopt in order to enhance our platform. What you'll do: Develop across our evolving technology stack - we're using Python, Java, Kubernetes, Apache Spark, Postgres, ArgoCD, Argo Workflow, Seldon, MLFlow and more. We are migrating into AWS cloud and adopting many services that are available in that environment. You will have the opportunity to … solutions. Strong analytical skills and desire to write clean, correct and efficient code. Sense of ownership, urgency and pride in your work. Experience with Python, Java, Docker, Kubernetes, Argo, Spark and AWS cloud services a plus. Exposure to Machine Learning practices a plus. We strive to create an accessible and inclusive experience for all candidates. If you need a More ❯

include: 3+ years experience in data engineering roles, delivering integrated data-driven applications Hands-on experience with Microsoft Fabric components (Pipelines, Lakehouse, Warehouses) Proficient in T-SQL and either Apache Spark or Python for data engineering Comfortable working across cloud platforms, with emphasis on Microsoft Azure Familiarity with REST APIs and integrating external data sources into applications More ❯

and streaming data pipelines Azure Purview or equivalent for data governance and lineage tracking Experience with data integration, MDM, governance, and data quality tools . Hands-on experience with Apache Spark, Python, SQL, and Scala for data processing. Strong understanding of Azure networking, security, and IAM , including Azure Private Link, VNETs, Managed Identities, and RBAC . Deep knowledge More ❯

Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯

Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯

Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯

to refine and monitor data collection systems using Scala and Java. Apply sound engineering principles such as test-driven development and modular design. Preferred Background Hands-on experience with Spark and Scala in commercial environments. Familiarity with Java and Python. Exposure to distributed data systems and cloud storage platforms. Experience designing data schemas and analytical databases. Use of AI More ❯

to refine and monitor data collection systems using Scala and Java. Apply sound engineering principles such as test-driven development and modular design. Preferred Background Hands-on experience with Spark and Scala in commercial environments. Familiarity with Java and Python. Exposure to distributed data systems and cloud storage platforms. Experience designing data schemas and analytical databases. Use of AI More ❯

between the data warehouse and other systems. Create deployable data pipelines that are tested and robust using a variety of technologies and techniques depending on the available technologies (Nifi, Spark) Build analytics tools that utilise the data pipeline to provide actionable insights into client requirements, operational efficiency, and other key business performance metrics. Complete onsite client visits and provide More ❯

between the data warehouse and other systems. Create deployable data pipelines that are tested and robust using a variety of technologies and techniques depending on the available technologies (Nifi, Spark) Build analytics tools that utilise the data pipeline to provide actionable insights into client requirements, operational efficiency, and other key business performance metrics. Complete onsite client visits and provide More ❯

experience of utilising strong communication and stakeholder management skills when engaging with customers Significant experience of coding (ideally Python) Experience with big data processing tools such as Hadoop or Spark Cloud experience; GCP specifically in this case, including services such as Cloud Run, Cloud Functions, BigQuery, GCS, Secret Manager, Vertex AI etc. Experience with Terraform Prior experience in a More ❯

analysis. Expertise in Python, with proficiency in ML and NLP libraries such as Scikit-learn, TensorFlow, Faiss, LangChain, Transformers and PyTorch. Experience with big data tools such as Hadoop, Spark, and Hive. Familiarity with CI/CD and MLOps frameworks for building end-to-end ML pipelines. Proven ability to lead and deliver data science projects in an agile More ❯

communication and stakeholder management skills when engaging with customers Significant experience of coding in Python and Scala or Java Experience with big data processing tools such as Hadoop or Spark Cloud experience; GCP specifically in this case, including services such as Cloud Run, Cloud Functions, BigQuery, GCS, Secret Manager, Vertex AI etc. Experience with Terraform Prior experience in a More ❯

part of an Agile engineering or development team Strong hands-on experience and understanding of working in a cloud environment such as AWS Experience with EMR (Elastic Map Reduce), Spark Strong experience with CI/CD pipelines with Jenkins Experience with the following technologies: SpringBoot, Gradle, Terraform, Ansible, GitHub/GitFlow, PCF/OCP/Kubernetes technologies, Artifactory, IaC More ❯

to solve complex client challenges Strong software engineering foundation in Python, JavaScript/TypeScript, SQL , and cloud platforms such as AWS, GCP, or Azure Familiarity with data technologies like Apache Spark or Databricks , and a structured, analytical approach to problem-solving If you're passionate about building AI-powered applications that positively impact millions of people and businesses … love to hear from you. Especially, if you know your way around all the following: Langchain, Lang graph, Agentic Frameworks, LLarma Python, JavaScript, TypeScript, AWS/GCP/Azure, Spark/Data Bricks, FastAPI/Flask, RAG, LLMs, GenAI, AI Solutions, Data Pipelines, Microservices, Solution Design, Software Development, Restful APIs, Chatbots, CI/CD Github Actions, AI Engineering, Unit More ❯

etc. Infrastructure as Code and CI/CD paradigms and systems such as: Ansible, Terraform, Jenkins, Bamboo, Concourse etc. Observability - SRE Big Data solutions (ecosystems) and technologies such as: Apache Spark and the Hadoop Ecosystem Excellent knowledge of YAML or similar languages The following Technical Skills & Experience would be desirable: Jupyter Hub Awareness RabbitMQ or other common queue More ❯

Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯

its native tech stack in designing and building data & AI solutions Experience with data modeling, ETL processes, and data warehousing Knowledge of big data tools and frameworks such as Spark, Hadoop, or Kafka More ❯

ETL processes is mandatory Experience building components for enterprise data platforms (data warehouses, Operational Data Stores, API access layers, file extracts, user queries) Hands-on experience with SQL, Python, Spark, Kafka Excellent communication skills, with proficiency in verbal and written English About this Job This role involves developing and maintaining real-time data processing pipelines for enterprise customer data. More ❯

ETL processes is mandatory Experience building components for enterprise data platforms (data warehouses, Operational Data Stores, API access layers, file extracts, user queries) Hands-on experience with SQL, Python, Spark, Kafka Excellent communication skills, with proficiency in verbal and written English About this Job This role involves developing and maintaining real-time data processing pipelines for enterprise customer data. More ❯

infrastructure from the ground up. Familiarity with AWS services like S3, EMR, and technologies like Terraform and Docker. Know the ins and outs of current big data frameworks like Spark or Flink, but this is not an absolute requirement - youre a quick learner! This role is open to individuals based in or willing to relocate to London. #J More ❯

working practices CI/CD tooling Scripting experience (Python, Perl, Bash, etc.) ELK (Elastic stack) JavaScript Cypress Linux experience Search engine technology (e.g., Elasticsearch) Big Data Technology experience (Hadoop, Spark, Kafka, etc.) Microservice and cloud native architecture Desirable skills Able to demonstrate experience of troubleshooting and diagnosis of technical issues. Able to demonstrate excellent team-working skills. Strong interpersonal More ❯

align tech strategy with business objectives and cost efficiency. Security & Compliance : Strong understanding of GDPR, API authentication, and observability. Big Data : Experience with data lakes, warehouses, and tools like Spark, Kafka, and Airflow. ETL Expertise : Ability to evaluate and optimize data ingestion and transformation pipelines. DevOps & CI/CD : Hands-on experience with Jenkins, GitHub Actions, Terraform, and CloudFormation. More ❯

align tech strategy with business objectives and cost efficiency. Security & Compliance : Strong understanding of GDPR, API authentication, and observability. Big Data : Experience with data lakes, warehouses, and tools like Spark, Kafka, and Airflow. ETL Expertise : Ability to evaluate and optimize data ingestion and transformation pipelines. DevOps & CI/CD : Hands-on experience with Jenkins, GitHub Actions, Terraform, and CloudFormation. More ❯