76 to 100 of 103 Apache Spark Jobs in the UK

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Spring, Spring Boot Experience working with DevOps methodologies and scalable CI/CD pipelines Microservices Architecture Exposure to Big Data technologies such as: Kafka, Spark, Hadoop etc Strong understanding of clean code principles Experience working with AWS Cloud Kotlin experience (preferred ...

with Git, Python, and Ansible Familiarity with containers (Docker, Kubernetes, LXC) Experience with monitoring tools and SQL databases Exposure to big data systems (e.g., Spark, HDFS, Airflow) Summary A great opportunity for someone who enjoys solving complex infrastructure challenges, automating systems, and working closely with engineering teams. Spectrum ...

with Git, Python, and Ansible Familiarity with containers (Docker, Kubernetes, LXC) Experience with monitoring tools and SQL databases Exposure to big data systems (e.g., Spark, HDFS, Airflow) Summary A great opportunity for someone who enjoys solving complex infrastructure challenges, automating systems, and working closely with engineering teams. Spectrum ...

team Exposure to sectors such as defence, public sector and consumer/retail Familiarity with modern data and AI tooling such as Python, Spark, Databricks, Azure cloud, and Generative AI platforms (experience with specific tools is helpful but not essential) A mix of technical, product and operational experience - comfortable ...

deploy scalable, secure data solutions using Azure Databricks, Data Factory and Data Lake Storage. Develop and optimise advanced data pipelines with Python, SQL, Spark/PySpark and Delta Lake. Champion strong data quality, governance and observability practices. Modernise legacy systems and support large-scale migrations into Azure. Provide technical … Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub ...

decisions Technical depth 10+ years in data engineering, data warehousing, or a senior data infrastructure role Expert‐level SQL and data manipulation skills using Spark, PySpark, or equivalent big data frameworks, with proven experience processing high‐volume, high‐velocity datasets Deep expertise in data lake and data warehousing technologies … Airflow or similar orchestration frameworks, with solid grounding in software design principles Strong Python proficiency; familiarity with streaming technologies such as Kafka or Apache Flink is a strong plus Hands‐on experience translating complex business requirements into robust data sourcing and modeling solutions Exposure to data mesh or data ...

data platforms Technical Expertise: Cloud platforms: Azure/AWS/GCP Data technologies: Data Lakes, Lakehouse, Data Mesh ETL/ELT pipelines, streaming (Kafka, Spark) Architecture patterns: Microservices, APIs, event-driven design Understanding of data governance, security, and compliance Key Responsibilities: 1. Enterprise Architecture & Strategy Define and drive enterprise ...

optimise scalable data pipelines (batch and real-time) Work primarily with Java , with exposure to Python and streaming frameworks such as Kafka, Flink, or Spark Develop event-driven architectures and streaming integrations Integrate data from multiple sources using AWS services (e.g. Kinesis, Lambda, Glue, MSK) Architecture & Data Management Contribute … experience in Data Engineering or large-scale data systems Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark (streaming focus is a strong advantage) Solid understanding of streaming concepts (event time, state, backpressure) Experience building ETL/ELT or real-time data ...

learning or LLM-based systems Exposure to enterprise data platforms or similar tools Frontend development experience (e.g., React) Familiarity with distributed computing frameworks (e.g., Spark) What’s on Offer Competitive compensation package, including equity Comprehensive benefits Hybrid working model Opportunity to build products and systems from the ground ...

Core programming for data analysis NumPy & Pandas – Data preparation and manipulation Matplotlib & Seaborn – Visualisation and exploratory analysis TensorFlow, SciPy, Scikit-learn – Machine learning fundamentals Spark – Scalable data processing Flask & Streamlit – Deploying APIs and interactive data applications Who This Program Is For Graduates or final-year students in STEM, computer ...

year; 1–3 years; 3–5 years; more than 5 years. Core technologies Akka (Actors, Streams, Persistence, Clustering, Projection, HTTP); ZIO; Play; Spark Libraries Circe; Cats; Cats Effect; Slick; Doobie; HTTP4S; Shapeless; Monix Databases RDBMS; MongoDB; Redis; Cassandra; Elastic Search Messaging Kafka; RabbitMQ Architecture and Cloud Microservices; DDD; Event ...

across complex engagements. What You’ll Do Lead design and implementation of enterprise-scale data pipelines and lakehouse architectures Provide technical leadership across Databricks, Spark and cloud data platforms Drive delivery of data engineering workstreams within Agile programmes Engage with client stakeholders to translate business objectives into technical solutions … architecture documentation, design artefacts and delivery roadmaps Must-Have Requirements 7+ years experience in data engineering or data platform development Strong experience with Databricks, Spark and distributed data processing Experience with Unity Catalog, Delta Lake and streaming architectures Advanced Python and Microsoft SQL Server development Experience designing enterprise data ...

Transformation You will develop and improve UCLH's data warehouse environments. Being involved in the transition from using an R-based data pipeline to Spark jobs running on a data platform. You will model and transform data from our Epic electronic health record system, coordinating closely with stakeholders … clinical safety and data governance standards. Desirable Experience of using R or Python Experience with modern data engineering tools and frameworks (e.g. dbt, snowflake, spark, etc.) Experience in an NHS informatics role or other healthcare information environment Skills and Abilities Essential In depth specialist knowledge of information analysis ...

About Quantiphi: Quantiphi is an award-winning, AI-First digital engineering and consulting company focused on delivering high-impact Services and Solutions that help organizations solve what truly matters. We partner with enterprises to reimagine ...

maintainable code Participating in code reviews, pairing, and team delivery Tech Stack & Environment Backend: Scala, Java (Python desirable) Frontend: HTML, CSS, JavaScript Frameworks: Play, Spark Data: Postgres, MongoDB, Hudi Cloud: AWS (S3, ECS, Aurora, Lambda) DevOps: Jenkins, Airflow, Terraform Monitoring: Kibana, Grafana Ways of working: Agile (Scrum/Kanban … Agile, multidisciplinary teams Strong analytical and problem-solving skills Clear communicator and collaborative team player Desirable Python, Docker, Airflow Big data tools (e.g. Spark, EMR) Experience with data platforms or risk/fraud systems How You'll Work Deliver value iteratively and early Collaborate closely across disciplines Write simple ...

handling edge cases, improving data quality and solving complex data challenges. The environment is modern and cloud based, with exposure to technologies such as Spark, Snowflake and BigQuery across AWS, GCP and Azure. You will also have the opportunity to support and mentor other engineers through collaboration, code reviews … Looking For Strong Python and SQL expertise gained within large scale production environments Experience delivering complex data transformation projects end to end Experience with Spark, Snowflake, BigQuery or similar technologies Exposure to cloud environments such as AWS, GCP or Azure Comfortable working with large, messy and evolving datasets Experience ...

A leading organisation undergoing major investment in its data and engineering capabilities is looking for a Senior Automation QA Engineer to strengthen quality across complex, distributed systems. This role sits at the intersection of software ...

A leading organisation undergoing major investment in its data and engineering capabilities is looking for a Senior Automation QA Engineer to strengthen quality across complex, distributed systems. This role sits at the intersection of software ...

systems, and modern cloud architectures . What youll be doing Design and build scalable data pipelines (batch & streaming) Work with technologies like Kafka, Flink, Spark, AWS services Develop real-time, event-driven data solutions Contribute to data architecture and modelling best practices Implement CI/CD pipelines and Infrastructure … deliver clean datasets What were looking for Strong programming skills in Java (preferred) or Python Experience with streaming technologies (Kafka/Flink/Spark) Knowledge of distributed systems and large-scale data processing Experience with cloud platforms (AWS preferred) Familiarity with DevOps tools and containerisation (Docker/Kubernetes ...

Developing and maintaining scalable data pipelines using Python - Automating manual Excel/VBA workflows into reliable, production-grade systems - Working with large datasets using Spark and Azure (Databricks/Synapse) - Supporting data modelling and analytics use cases (Crossing over to more Data Science driven work) - Driving best practices including … code reviews - Collaborating with stakeholders to deliver actionable data solutions What they re looking for: - Strong Python and SQL skills - Experience or exposure to Spark, big data tools, or cloud platforms (ideally Azure) What s on offer: - The chance to join a business at an exciting stage of growth ...

Developing and maintaining scalable data pipelines using Python - Automating manual Excel/VBA workflows into reliable, production-grade systems - Working with large datasets using Spark and Azure (Databricks/Synapse) - Supporting data modelling and analytics use cases (Crossing over to more Data Science driven work) - Driving best practices including … code reviews - Collaborating with stakeholders to deliver actionable data solutions What they’re looking for: - Strong Python and SQL skills - Experience or exposure to Spark, big data tools, or cloud platforms (ideally Azure) What’s on offer: - The chance to join a business at an exciting stage of growth ...

match rates, reduce false positives/negatives, and support downstream fraud and KYC models. Build and maintain scalable data pipelines and feature stores in Spark/PySpark (or Scala), including data normalization, deduplication, and feature computation across large PII datasets in AWS/Databricks environments. Lead A/… preferred. Strong proficiency in Python (preferred) or Scala, including experience with ML libraries such as scikit‐learn, XGBoost, TensorFlow or PyTorch. Extensive experience with Spark or PySpark and distributed data systems (e.g., AWS EMR, Databricks) working on very large, messy datasets. Deep understanding of supervised and unsupervised learning, feature ...

Design, build, and maintain scalable data pipelines and workflows to support analytics, fraud detection, model development, and ongoing data monitoring (e.g., using Spark, Airflow, or similar distributed systems). Leverage and build agentic AI and LLM-powered systems to automate data exploration, anomaly detection, vendor evaluation, and investigative workflows … both batch and near‐real‐time use cases. Experience building and maintaining data pipelines and workflows in distributed or large‐scale environments (e.g., Spark, Airflow, Databricks, or similar technologies). Demonstrated ability to evaluate and work with third‐party data vendors or external datasets, including designing tests for data ...

Educated to degree level in a relevant IT subject. Expertise in designing and implementing data pipelines using Azure services (Azure Data Factory, Data storage), Spark and Databricks. Expertise in data modelling, database design and designing enterprise data architecture . Azure Data Stack SQL server, NoSQL, Nanobricks, ADT, Spark ...