26 to 50 of 50 Apache Spark Jobs in the UK excluding London

driven decision making. What youll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across modern ...

driven decision making. What you'll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across ...

backend roles Strong Python; experience with distributed data systems Relational DB experience (PostgreSQL preferred) NoSQLExposure to vector databases (Qdrant, Milvus, pg vector) Bonus: Spark/Airflow/Kafka, cloud (GCP/AWS/Azure), AI/ML platform experience, early-stage startup bac kground Why Join Build the data ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

Work with container technologies Core Data Engineering Strong programming proficiency in Java or Python. Hands-on experience with at least one of: Kafka, Flink, Spark Solid understanding of stream processing concepts Understanding of software engineering best practices: testing, design patterns, CI/CD, Git. Experience building ETL/ ...

platforms that enable real‐time and batch data processing at scale. Responsibilities Drive platform capability improvements – continuously enhance distributed data platforms using knowledge of Spark, Databricks, Kafka, Pulsar, Cassandra, and object stores to maintain best‐in‐class capabilities. Enable application data consumption – understand how applications produce and consume data … structures, partitioning strategies, consistency models, and performance optimisation, as well as OLTP databases such as PostgreSQL and MySQL. Distributed compute frameworks – strong experience with Apache Spark and distributed data processing, including optimisation, performance tuning, and large‐scale batch/streaming workloads. Cloud‐native data platforms – demonstrated experience with ...

ownership Technical Application ownership Qualifications Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

services, capital markets or FinTech. Additional AWS services such as SQS, SNS, Step Functions, EventBridge or Secrets Manager. Big data technologies including DuckDB, pandas, Spark, Databricks or Snowflake. AI/ML technologies including vector databases, semantic search or LLM orchestration frameworks. Continuous delivery and modern development workflows. Data science ...

experience applying them with packages like Pandas, sci-kit-learn, SciPy) is highly beneficial Experience processing big data, ideally in a Hadoop/Spark environment, would be beneficial Proven experience implementing CI/CD and DevOps practices, with the ability to improve deployment reliability, testing standards, and release processes ...

partner management and ability to influence across teams and organisations. Continuous learning and improvement mentality. Desired Experience with big data technologies (e.g. Hadoop, Hive, Spark). Experience with generative AI, LLMs, or retrieval-augmented generation (RAG). Exposure to Agentic AI concepts, including autonomous agents, tool use, and orchestration ...

Nice to have: Experience with CI/CD pipelines (GitLab, Jenkins) Exposure to data tooling (Airflow, NiFi, Elasticsearch) Experience with Cloudera/Hadoop ecosystems (Spark, Hive, HDFS) Why join? Work on modern data platforms at scale Collaborate with data scientists and engineering teams Opportunity to shape platform architecture ...

platforms (AWS, Azure, or GCP). Distributed Systems Mastery: Deep technical capability in structuring data platforms, data pipelines, and high‐throughput distributed systems (like Spark or Kafka) optimized for performance and scalability. Cross‐Functional Technical Leadership: Exceptional ability to bridge the gap between deep technical engineering and strategic business ...

Strong experience with MLOps, production ML systems, model lifecycle management, and monitoring. Experience working with large-scale data systems and distributed computing frameworks (e.g. Spark, Hadoop). Knowledge of experimental design, scientific methodology, and analysis. Strong partner management and confirmed ability to influence large organisations without direct authority. Demonstrated ...

unstructured workloads at scale - Integrate multiple sources into modern cloud platforms - Collaborate across engineering and delivery on mission-critical programmes **The environment** Python, SQL, Spark, Java or Scala · AWS, Azure or GCP · modern distributed and cloud-native architectures **What we need** - Active DV clearance essential - Production-scale pipeline experience ...

Weaviate or pgvector Docker, Kubernetes, ECS or containerised deployments Infrastructure as Code tools including Terraform or CloudFormation Large-scale data processing technologies such as Spark or Dask Experience working within regulated or highly secure environments Knowledge of data governance and compliance frameworks Apply for This Machine Learning Engineer Opportunity ...

with large-scale and unstructured datasets. Python development experience, including technologies such as PySpark, Pandas and PyArrow. Expertise in distributed data processing frameworks, particularly Apache Spark. Experience building and maintaining ETL pipelines using technologies such as Apache Airflow, AWS Step Functions or Apache NiFi. Cloud platform experience ...

kubernetes) Experience with Key value stores Experience in git and basic Unix Commands You might also have (Desirable) Experience with large data processing technologies (Apache Spark) Other helpful information Hybrid Working: At EMBL-EBI we are pleased to offer hybrid working options for all our employees. A dedicated ...

help deliver robust, secure and high-performing data products. What youll be doing: Design and build scalable data pipelines using Python, SQL, Spark and Databricks. Develop, deploy and optimise machine learning models, including NLP and generative AI solutions. Build data ingestion pipelines across batch and streaming environments. Design data … visualisations using Power BI or Tableau. Mentor junior team members and promote engineering best practices. What you need: Commercial experience with Python, SQL, Spark and modern data engineering frameworks. Experience deploying machine learning models into production. Strong knowledge of cloud platforms such as Azure, AWS or GCP. Experience with ...

like ER/Studio, ERwin, or Sparx EA (preferred). Experience reverse engineering models from existing databases. Proficiency in distributed data processing frameworks (e.g., Spark, Flink, Hadoop). Experience with API management and gateway tools and services. Experience of RESTful APIs for ingesting and exposing data. Proven track record … with data governance, quality, lineage, and security practices. Experience with real-time/streaming data technologies, data Ingestion/ETL (e.g., Apache Kafka, Apache NiFi, Kinesis, Pub/Sub). Experience in, or knowledge of, DevSecOps Tooling and Processes. Self-starter with the ability to appropriately prioritise ...

with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well ...

Data Engineer Location: London (Hybrid) Salary: Up to £75,000 + Bonus + Benefits Technology: SQL, Python, Spark - Azure, AWS, GCP - ETL/ELT - Data Governance The Opportunity Are you a Data Engineer looking for an opportunity to work on large-scale data transformation programmes whilst helping shape enterprise … data lineage. Support Data Governance initiatives, ensuring data security, privacy and compliance standards are maintained. Work with large and complex datasets using SQL, Python, Spark, Hadoop and modern data technologies. Collaborate with technical and business stakeholders within Agile delivery environments. Support data architecture and data modelling activities across enterprise ...

architecture and deployment. Key Responsibilities Data Pipeline Development Build, optimize, and maintain scalable data pipelines using Java, plus exposure to Python, Flink, Kafka, or Spark click apply for full job details ...

experience owning and operating production data infrastructure at significant scale. Proven capability delivering large-scale streaming data solutions using technologies such as Kafka, Flink, Spark Streaming, Dataflow, or similar. Demonstrated success leading end-to-end migrations of live production systems. Strong knowledge of cloud platforms and distributed data architectures. … with data scientists, analysts, and business stakeholders. Strong understanding of data platform reliability, performance, scalability, and cost optimisation. Experience with technologies such as ClickHouse, Apache Iceberg, BigQuery, dbt, Elasticsearch, Druid, GCP, or AWS is highly desirable. If you're an experienced Senior Data Engineer looking to lead complex data ...

Data Architect Inside IR35 up to £600+ Nottingham Hybrid Summary We're undertaking a fast paced data transformation into Databricks at E.ON Next using best practice data governance and architectural principles, and we are growing ...