51 to 75 of 76 Apache Spark Jobs in London

ensure data integrity and reliability. Optimise data workflows for performance, cost-efficiency, and maintainability using tools such as Azure Data Factory, AWS Data Pipeline for Data Orchestration, Databricks, or Apache Spark. Support the integration of data into visualisation platforms (e.g. Power BI, ServiceNow) and other analytical environments. Ensure compliance with data governance, security, and privacy policies. Document data architecture More ❯

and ETL/ELT processes. Proficiency in AWS data platforms and services. Solid understanding of data governance principles (data quality, metadata, access control). Familiarity with big data technologies (Spark, Hadoop) and distributed computing. Advanced SQL skills and proficiency in at least one programming language (Python, Java). Additional Requirements Immediate availability for an October start. Must be UK More ❯

to create, test and validate data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What’s in it for you? Base More ❯

to create, test and validate data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What’s in it for you? Base More ❯

days a week – based business need. To Be Considered: Demonstrable expertise and experience working on large-scale Data Engineering projects Strong experience in Python/PySpark, Databricks & Apache Spark Hands on experience with both batch & streaming pipelines Strong experience in AWS and associated tooling (Eg, S3, Glue, Redshift, Lambda, Terraform etc) Experience designing Data Engineering platforms from scratch More ❯

days a week – based business need. To Be Considered: Demonstrable expertise and experience working on large-scale Data Engineering projects Strong experience in Python/PySpark, Databricks & Apache Spark Hands on experience with both batch & streaming pipelines Strong experience in AWS and associated tooling (Eg, S3, Glue, Redshift, Lambda, Terraform etc) Experience designing Data Engineering platforms from scratch More ❯

and at home, there's nothing we can't achieve. BASIC QUALIFICATIONS - 10+ years of technical specialist, design and architecture experience - 10+ years of database (eg. SQL, NoSQL, Hadoop, Spark, Kafka, Kinesis) experience - 10+ years of consulting, design and implementation of serverless distributed solutions experience - Australian citizen with ability to obtain security clearance. PREFERRED QUALIFICATIONS - AWS Professional level certification More ❯

Preferred: Experience in front-office roles or collaboration with trading desks Familiarity with financial instruments across asset classes (equities, FX, fixed income, derivatives) Experience with distributed computing frameworks (e.g., Spark, Dask) and cloud-native ML pipelines Exposure to LLMs, graph learning, or other advanced AI methods Strong publication record or open-source contributions in ML or quantitative finance Please More ❯

Preferred: Experience in front-office roles or collaboration with trading desks Familiarity with financial instruments across asset classes (equities, FX, fixed income, derivatives) Experience with distributed computing frameworks (e.g., Spark, Dask) and cloud-native ML pipelines Exposure to LLMs, graph learning, or other advanced AI methods Strong publication record or open-source contributions in ML or quantitative finance Please More ❯

equivalent UCAS points (please ensure A-Level grades are included on your CV). Basic scripting knowledge in Python or Bash Excellent customer-facing skills You have a sales spark - while this role isn't a focussed sales role, this is required due to the nature of the role A motivated self-starter with a problem-solving attitude Strong More ❯

be required in the role; we are happy to support your learning on the job, but prior experience is a plus: Experience with large-scale data processing frameworks (e.g., Spark, Flink). Experience with time series analysis, anomaly detection, or graph analytics in a security context. Proficiency in data visualization tools and techniques to effectively communicate complex findings. A More ❯

assessments and predictive models. Optimize models for performance, scalability, and accuracy. Qualifications: Deep knowledge of neural networks (CNNs, RNNs, LSTMs, Transformers). Strong experience with data tools (Pandas, NumPy, Apache Spark). Solid understanding of NLP algorithms. Experience integrating ML models via RESTful APIs. Familiarity with CI/CD pipelines and deployment automation. Strategic thinking around architecture and More ❯

the latest tech, serious brain power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew of sharp analytical minds? You'll More ❯

the latest tech, serious brain power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew of sharp analytical minds? You'll More ❯

the latest tech, serious brain power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything-spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew of sharp analytical minds? You'll More ❯

areas: Data integrity, scalability, and cost efficiency Managing complex data workflows and ML pipelines Ensuring model quality and performance Partnering cross-functionally across engineering, science, and product Tech Stack: Spark, Cassandra, Redshift AI/ML Use Cases: Predictive click models, data quality optimization, GenAI search applications What You'll Bring 5-10 years' experience overall, with 3+ years in More ❯

areas: Data integrity, scalability, and cost efficiency Managing complex data workflows and ML pipelines Ensuring model quality and performance Partnering cross-functionally across engineering, science, and product Tech Stack: Spark, Cassandra, Redshift AI/ML Use Cases: Predictive click models, data quality optimization, GenAI search applications What You'll Bring 5-10 years' experience overall, with 3+ years in More ❯

on experience across AWS Glue, Lambda, Step Functions, RDS, Redshift, and Boto3. Proficient in one of Python, Scala or Java, with strong experience in Big Data technologies such as: Spark, Hadoop etc. Practical knowledge of building Real Time event streaming pipelines (eg, Kafka, Spark Streaming, Kinesis). Proven experience developing modern data architectures including Data Lakehouse and Data … and data governance including GDPR. Bonus Points For Expertise in Data Modelling, schema design, and handling both structured and semi-structured data. Familiarity with distributed systems such as Hadoop, Spark, HDFS, Hive, Databricks. Exposure to AWS Lake Formation and automation of ingestion and transformation layers. Background in delivering solutions for highly regulated industries. Passion for mentoring and enabling data More ❯

affordable charging infrastructure that is accessible to all, particularly the 40% of UK households without private driveways that rely on public charging. We believe our employees are the electric spark igniting our success. Were dedicated to our culture, and ensure that every individual feels charged up and empowered to accelerate their career which is why we have been recognised More ❯

be on designing and maintaining the data pipelines that feed large-scale ML and research workflows. Day-to-day responsibilities include: Building and maintaining data pipelines using Python, SQL, Spark, and Google Cloud technologies (BigQuery, Cloud Storage). Ensuring pipelines are robust, reliable, and optimised for AI/ML use cases. Developing automated tests, documentation, and monitoring for production … best practices, and continuously improving performance and quality. Tech Stack & Skills Core Skills: Strong experience with Python and SQL in production environments Proven track record developing data pipelines using Spark, BigQuery, and cloud tools (preferably Google Cloud) Familiarity with CI/CD and version control (git, GitHub, DevOps workflows) Experience with unit testing (e.g., pytest) and automated quality checks More ❯

team leadership and upskilling responsibilities. Key Responsibilities Build and maintain Databricks Delta Live Tables (DLT) pipelines across Bronze ? Silver ? Gold layers, ensuring quality, scalability, and reliability. Develop and optimise Spark (PySpark) jobs for large-scale distributed processing. Design and implement streaming data pipelines with Kafka/MSK, applying best practices for late event handling and throughput. Use Terraform and … role) Mentor and upskill engineers, define coding standards, and embed engineering excellence across the team. What's Expected Proven experience delivering end-to-end data pipelines in Databricks and Spark environments. Strong understanding of data modelling, schema evolution, and data contract management. Hands-on experience with Kafka, streaming architectures, and real-time processing principles. Proficiency with Docker, Terraform, and More ❯

optimising end-to-end data pipelines using Azure Databricks, PySpark, ADF, and Delta Lake Implementing a medallion architecture - from raw to enriched to curated Working with Delta Lake and Spark for both batch and streaming data Collaborating with analysts to validate and refine datasets for reporting Applying CI/CD and DevOps best practices (Git, Azure DevOps) Enforcing data … governance using Azure Purview and Unity Catalog Optimising Spark jobs and SQL queries for performance and cost efficiency Exploring emerging tech like Kafka/Event Hubs and Knowledge Graphs What they're looking for: A strong communicator - someone who can build relationships across technical and business teams Hands-on experience building pipelines in Azure using Databricks, ADF, and PySpark More ❯

in real-time and batch data. Build and scale high-volume data pipelines handling billions of daily events across distributed systems. Work with cutting-edge tech including Java, Kafka, Spark, Kubernetes, Clickhouse, Snowflake, Redis . Collaborate with quants, data scientists, and compliance experts to improve surveillance strategies. Contribute to system architecture for low-latency monitoring and high-throughput processing … skills in Java (or similar: Kotlin, C#, C++) ; Python is a plus. Proven experience with real-time, low-latency systems and distributed data pipelines . Hands-on with Kafka, Spark, K8s, Clickhouse, Snowflake, and modern data architectures . Solid foundations in algorithms, system design, and optimisation . Curious, proactive, and comfortable working in a fast-moving, scaling environment. Why More ❯

Senior Software Engineer - Remote (UK) Location: Remote (UK-based)/very occasional travel to London A fast-growing global technology company specialising in data-driven B2B solutions is seeking a contract Senior Software Engineer to join its high-performing engineering More ❯

learning methods and machine learning Experience in building machine learning models for business application Experience in applied research PREFERRED QUALIFICATIONS Experience with modeling tools such as R, scikit-learn, Spark MLLib, MxNet, Tensorflow, numpy, scipy etc. Experience with large scale distributed systems such as Hadoop, Spark etc. PhD Amazon is an equal opportunities employer. We believe passionately that More ❯