1 to 25 of 432 Apache Spark Jobs in the UK

detect threats, analyze patterns, and respond to incidents in real-time.You'll be designing and developing highly scalable streaming services using technologies like Apache Kafka, Apache Flink, Apache Storm and Apache Spark in Java, Python and Go. You'll design and implement data pipelines that … Design, develop, and maintain streaming microservices in Java and Go that provide real-time data processing capabilities to internal teamsBuild and optimize Apache Flink and Apache Spark based data pipelines running on Kubernetes infrastructureDesign event-driven systems leveraging Kafka for high-throughput, low-latency message streamingTake ...

develop, and maintain end-to-end data pipelines using Ab Initio (GDE, PDL, Conduct>IT, Express>IT).Build scalable batch and streaming workloads using Apache Spark (PySpark/Scala).Develop and orchestrate data ingestion flows using Apache NiFi for real-time and near-real-time use cases.Optimize …/ELT jobs for scalability, throughput, and resource efficiency.Perform bottleneck analysis and tuning for Spark jobs, NiFi processors, and Ab Initio graphs.Demos and knowledge transfers to stakeholders around product developments.Spread your experience and knowledge to junior engineers.Improve automated testing and deployments using CI/CD pipelines.Expectation to line manage ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities.Team Leadership & Delivery ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

product managers, to understand data requirements and deliver high-quality solutions Architect and implement data ingestion, transformation, and storage processes using tools such as Apache Spark, Azure Data Factory, and other similar technologies Optimize data pipeline performance, ensuring data accuracy, reliability, and timely delivery Mentor and guide junior … Engineering, or a related field 7+ years of experience in data engineering, with at least 3 years in a leadership role Strong proficiency in Apache Spark, Azure Data Factory, and other data pipeline tools Experience with cloud platforms such as Azure, AWS, or GCP Solid understanding of data ...

Databricks-based Lakehouse that drives real-time data, analytics, and reporting across the Tote.You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast … deliver scalable, secure, and governed data solutions under Unity Catalogue.What you’ll be doing:You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop ...

applications, working closely with data engineers, analysts, and platform teams. Day to day duties Developing and maintaining data pipelines and processing jobs using Apache Spark and Python Orchestrating workflows with Apache Airflow Provisioning and managing cloud infrastructure on AWS using Terraform (Infrastructure as Code) Containerising applications … teams in an Agile environment, contributing to design reviews, code reviews, and documentation Essential Skills & Experience Strong hands-on experience with Python Experience with Apache Spark for large-scale data processing Practical knowledge of AWS services (eg S3, Lambda, EC2, Glue, RDS - adjust as relevant) Experience writing ...

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

databases utilised in modern AI applications. Strong Python development skills. Data & Infrastructure: Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure: Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases: PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Ecosystem: Pandas Polars ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain). Hands-on experience working with Distributed Spark based systems Experience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external … clients across a variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred [Preferred] Experience working with Apache Spark to process large-scale distributed datasets Communication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences. Core ...

frameworks, tools and applications that make that data available to other teams and systems.Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

enjoys working in a data-rich environment — not a traditional pipeline-focused data engineer. You’ll design and build scalable data systems using Python, Spark and AWS, while also shaping engineering standards, guiding other engineers, and helping the team make sound technical and architectural decisions. This is an opportunity … properly — this is the role for you. WHAT YOU'LL BE DOING:Build & deliver Design and develop scalable, testable data pipelines using Python and Apache Spark Build and orchestrate data workflows using AWS services (e.g. Glue, EMR Serverless, Lambda, S3) Contribute to the evolution of a modern lakehouse ...

peers on technical direction within your teamRequired Skills and ExperienceStrong analytic skills related to working with unstructured datasetsPython (PySpark, Pandas, PyArrow)Distributed data processing (Apache Spark)Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi)Cloud services (AWS, Azure or GCP)Messaging/Streaming (Kafka ...

peers on technical direction within your teamRequired Skills and ExperienceStrong analytic skills related to working with unstructured datasetsPython (PySpark, Pandas, PyArrow)Distributed data processing (Apache Spark)Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi)Cloud services (AWS, Azure or GCP)Messaging/Streaming (Kafka ...

peers on technical direction within your teamRequired Skills and ExperienceStrong analytic skills related to working with unstructured datasetsPython (PySpark, Pandas, PyArrow)Distributed data processing (Apache Spark)Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi)Cloud services (AWS, Azure or GCP)Messaging/Streaming (Kafka ...

data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc Experience developing end‐to‐end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam. Expert understanding of when to use a range of data storage technologies including relational/non‐relational, document ...

ready data processing solutions using BigQuery, Pub/Sub, Dataflow and DataprocExperience developing end-to-end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam.Expert understanding of when to use a range of data storage technologies including relational/non-relational, document ...