101 to 125 of 436 Apache Spark Jobs in the UK

intellectually rewarding, involving large datasets, performance-critical pipelines, and precision engineeringWe value strong individual contributors — this is a hands-on role where your Scala Spark expertise will directly shape the platform's futureWe are a team that is enthusiastic about what we are building — there is a genuine drive … have the opportunity to contribute to architectural decisions, not just execute themRole Overview/What will you do:Build and optimise large-scale Scala Spark data pipelines on Databricks, focusing on the reconciliation domain — writing clean, modular, and production-ready codeDrive modernisation efforts by refactoring legacy Hadoop-based pipelines ...

BigQuery, Dataflow, Dataproc, Cloud Composer (Airflow), Pub/Sub, Cloud Storage, Bigtable, and Cloud SQL. Proficiency in Python and SQL, with experience in Spark or Apache Beam. Experience designing data warehouse and data lake architectures, including data modeling. Working experience with Terraform and CI/CD tooling such ...

analytics and AI/ML at enterprise scale.* Develop and optimise batch and streaming ETL/ELT pipelines using Azure Data Factory, PySpark, SQL, Apache Spark and Airflow, integrated with modern tooling like dbt and Git-based CI/CD.* Implement data cataloguing, lineage tracking and governance frameworks … data engineering, including designing and delivering data lake or lakehouse architectures in an enterprise cloud environment, ideally Microsoft Azure.* Hands-on expertise with Databricks, Spark, PySpark, SQL and core Azure Data Services (Data Lake Storage, Synapse Analytics, Data Factory).* Experience with data governance and cataloguing tools such ...

working with cloud‐native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such … Glue, EMR Serverless, Lambda, and S3. Contribute to the evolution of lakehouse architecture leveraging Apache Iceberg. Apply Software Engineering Best Practices Use version control, CI/CD pipelines, automated testing, and modular code principles. Participate in pair programming, code reviews, and architectural design sessions. Data Quality & Observability Build monitoring ...

product lifecycle. Hands‐on experience with large‐scale data processing and machine learning in production environments. Strong expertise in distributed computing frameworks such as Apache Spark or similar technologies. Solid understanding of machine learning workflows, including model development, deployment, and inference. Strong knowledge of modern data architectures, including ...

taking customers from EDW to the full capabilities of the Data Intelligence Platform. Demonstrated competence in the Lakehouse architecture including hands-on experience with Apache Spark, Python and SQL Excellent communication skills; both written and verbal Public Cloud Architecting: Hands-on experience designing and implementing big data architectures ...

gateways for AI applications Infrastructure as Code experience with Terraform, Ansible, or other IaC tools Experience building and managing data pipelines using tools like Apache Spark AI safety and governance experience with model evaluation, LLM judges, bias detection, and responsible AI practices Familiarity with statistical fundamentals and running ...

Experience integrating multiple business systems, applications and third-party data sources using cloud-based technologies. Desirable experience with Azure Data Factory, Microsoft Purview, Python, Spark, DataOps, DevOps, AI platforms or large-scale cloud modernisation projects. ...

Experience integrating multiple business systems, applications and third-party data sources using cloud-based technologies. Desirable experience with Azure Data Factory, Microsoft Purview, Python, Spark, DataOps, DevOps, AI platforms or large-scale cloud modernisation projects. ...

bricks, Data Lake, Logic Apps and Data Factory Experience migrating or transforming legacy solutions to Public Cloud desirable Understanding of Big Data technologies (Hadoop, Spark) Familiarity with infrastructure considerations for data and system integration Broad knowledge of the design and implementation of applications' build, release, deployment, and configuration activities ...

supporting or delivering migrations from Azure Synapse to Microsoft Fabric Advanced SQL: Strong SQL and T-SQL development, optimisation and troubleshooting skills. Python and Spark: Experience building data processing and transformation solutions using Python and Spark. Data engineering fundamentals: Strong understanding of ETL/ELT, data warehousing, dimensional modelling ...

with Microsoft Azure a plus: Azure SQL Database, Analysis Services, Databricks, Data Lake, Logic Apps and Data Factory Understanding of Big Data technologies (Hadoop, Spark) Able to create rich visualisations that appropriately represent the underlying datasets. Review and align the reporting attributes against the Enterprise and conceptual data models ...

culture where people feel safe to challenge ideas, learn, and grow. Nice to have Hands-on familiarity with the modern data stack — for example, Spark, Flink, Kafka, Debezium, Airflow, dbt, or similar — and with cloud platforms (AWS/GCP or equivalent) and modern DevOps practices. Experience leading teams that ...

data analytics, or data science, with the ability to work effectively with various data types and sources. Experience using big data technologies (e.g. Hadoop, Spark, Hive) and database management systems (e.g. SQL and NoSQL). Graph Database Expertise : Deep understanding of graph database concepts, data modeling, and query languages ...

this role leads the engineering of cloud-native data services for the firm’s strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS. The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled. As platform landscape evolved towards cloud-native, AI enabled ...

support comprehensive data analysis and reporting. Implement and uphold best practices in data governance, security, and compliance, using tools like Azure Purview, UnityCatalog, and Apache Atlas to maintain data integrity and trust. Ensure data quality and integrity through meticulous attention to detail and rigorous QA processes, continually refining … ease. Understanding of Data Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using ...

services, a good understanding of cloud architecture, and proficiency with Terraform for provisioning and managing cloud infrastructure Experience building streaming data pipelines using Apache Flink, Apache Kafka, Spark Structured Streaming, or similar technologies Knowledge of modern data lakehouse architectures and ecosystem such as Kafka, Flink, Spark … with a SaaS or FinTech company Technologies we use: AWS Services: MSK, EC2, VPC, IAM, S3, SQS, Managed Flink, EKS Data Lakehouse: Kafka, Flink, Spark, Databricks, DBT, Debezium #J-18808-Ljbffr ...

cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

schemas for both structured and unstructured data with a snowflake environment Enhance cloud data capabilities, working with AWS S3, Azure Data Lake, and Apache Spark Collaborate with cross-functional teams to ensure data availability for advanced analytics and reporting Automate workflows, implement CI/CD pipelines, and maintain ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

platform. Languages & Core - Advanced proficiency in Python (OOP, typing, design patterns, testing frameworks) and advanced analytical SQL. Distributed Computing - Deep hands-on knowledge of Apache Spark/PySpark, including distributed query optimization, partition tuning, memory management, and debugging skewed datasets. ML & MLOps - Strong experience with MLflow (Tracking, Model ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

platforms using AWS, Azure or Google Cloud. Expertise in data modelling, SQL, NoSQL and modern database technologies. Experience with big data technologies, such as Apache Spark, Hadoop or distributed computing frameworks. Programming knowledge in Python, Scala, Java or similar languages. Experience architecting AI and machine learning solutions, including ...

designing data platforms using AWS, Azure or Google Cloud.Expertise in data modelling, SQL, NoSQL and modern database technologies.Experience with big data technologies, such as Apache Spark, Hadoop or distributed computing frameworks.Programming knowledge in Python, Scala, Java or similar languages.Experience architecting AI and machine learning solutions, including deploying models ...