1 to 25 of 37 Permanent Apache Spark Jobs in London

while staying close to the code. Perfect if you want scope for growth without going "post-technical." What you'll do Design and build modern data platforms using Databricks, Apache Spark, Snowflake, and cloud-native services (AWS, Azure, or GCP). Develop robust pipelines for real-time and batch data ingestion from diverse and complex sources. Model and … for Solid experience as a Senior/Lead Data Engineer in complex enterprise environments. Strong coding skills in Python (Scala or functional languages a plus). Expertise with Databricks, Apache Spark, and Snowflake (HDFS/HBase also useful). Experience integrating large, messy datasets into reliable, scalable data products. Strong understanding of data modelling, orchestration, and automation. Hands More ❯

MySQL) to modern NoSQL solutions (e.g., MongoDB, Cassandra). Focus on strategies that enhance data accessibility, integrity, and performance. Big Data Processing & Analytics: Utilise big data frameworks such as Apache Spark and Apache Flink to address challenges associated with large-scale data processing and analysis. These technologies are crucial for managing vast datasets and performing complex data … upgrading skills and achieving certifications in the technologies Simple Machines implements across its client base. Ideal Skills and Experience Core Data Engineering Tools & Technologies: Demonstrates proficiency in SQL and Spark, and familiarity with platforms such as Databricks and Snowflake. Well-versed in various storage technologies including AWS S3, Google Cloud BigQuery, Cassandra, MongoDB, Neo4J, and HDFS. Adept in pipeline … orchestration tools like AWS Glue, Apache Airflow, and dbt, as well as streaming technologies like Kafka, AWS Kinesis, Google Cloud Pub/Sub, and Azure Event Hubs. Data Storage Expertise: Knowledgeable in data warehousing technologies like BigQuery, Snowflake, and Databricks, proficient in managing various data storage formats including Parquet, Delta, ORC, Avro, and JSON to optimise data storage and More ❯

modelling tools, data warehousing, ETL processes, and data integration techniques. · Experience with at least one cloud data platform (e.g. AWS, Azure, Google Cloud) and big data technologies (e.g., Hadoop, Spark). · Strong knowledge of data workflow solutions like Azure Data Factory, Apache NiFi, Apache Airflow etc · Good knowledge of stream and batch processing solutions like Apache Flink, Apache Kafka/· Good knowledge of log management, monitoring, and analytics solutions like Splunk, Elastic Stack, New Relic etc Given that this is just a short snapshot of the role we encourage you to apply even if you don't meet all the requirements listed above. We are looking for individuals who strive to make an impact More ❯

two of the following: Python, SQL, Java Commercial experience in client-facing projects is a plus, especially within multi-disciplinary teams Deep knowledge of database technologies: Distributed systems (e.g., Spark, Hadoop, EMR) RDBMS (e.g., SQL Server, Oracle, PostgreSQL, MySQL) NoSQL (e.g., MongoDB, Cassandra, DynamoDB, Neo4j) Solid understanding of software engineering best practices - code reviews, testing frameworks, CI/CD More ❯

Skills: Proven expertise in designing, building, and operating data pipelines, warehouses, and scalable data architectures. Deep hands-on experience with modern data stacks. Our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform. Experience with similar technologies is essential. Coaching & Growth Mindset: Passion for developing More ❯

have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands on coding experience, such as More ❯

platform. Candidate Profile: Proven experience as a Data Engineer, with strong expertise in designing and managing large-scale data systems. Hands-on proficiency with modern data technologies such as Spark, Kafka, Airflow, or dbt. Strong SQL skills and experience with cloud platforms (Azure preferred). Solid programming background in Python, Scala, or Java. Knowledge of data warehousing solutions (e.g. More ❯

have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools (e.g. Tableau, PowerBI, Qlik More ❯

Data Software Engineer/Developer (Python Spark Azure) London/WFH to £100k Are you a data centric Software Engineer with strong Python coding skills? You could be progressing your career in a senior, hands-on Data Software Engineer role at a scaling, global technical services company as they look to expand their product offerings with a new SaaS … technical challenges, you'll be collaboratively problem solving as part of an Agile development using a range of technology to create data pipelines with a focus on Python and Spark; you'll be working with Azure, ETL pipelines and CI/CD, ingesting and analysing terabytes of data with varying structures from a range of sources. You'll be … London office. About you : You have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI, Django, Airflow, Kafka, ETL More ❯

Terraform, CloudFormation) and CI/CD workflows · If you have previous exposure to geospatial data, that would be advantageous but is not a requirement for the position. · Familiarity with Apache Spark or Databricks · Excellent communication and collaboration skills Benefits About Prevail Partners Prevail Partners delivers strategic advice, intelligence, specialist capabilities, and managed services to clients ranging from governments More ❯

within the Insurance industry Strong proficiency in programming languages such as Python, Java, or Scala. Experience with cloud platforms like Azure. Knowledge of big data technologies such as Hadoop, Spark, or Kafka. Proficiency in SQL and database management systems. Familiarity with data warehousing concepts and tools. Ability to work collaboratively with cross-functional teams. A solid understanding of data More ❯

within the Insurance industry Strong proficiency in programming languages such as Python, Java, or Scala. Experience with cloud platforms like Azure. Knowledge of big data technologies such as Hadoop, Spark, or Kafka. Proficiency in SQL and database management systems. Familiarity with data warehousing concepts and tools. Ability to work collaboratively with cross-functional teams. A solid understanding of data More ❯

e.g., PostgreSQL, DuckDB). Experience with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic modelling, summarisation), statistical analysis, and More ❯

communication and stakeholder management skills when engaging with customers Significant experience of coding in Python and Scala or Java Experience with big data processing tools such as Hadoop or Spark Cloud experience; GCP specifically in this case, including services such as Cloud Run, Cloud Functions, BigQuery, GCS, Secret Manager, Vertex AI etc. Experience with Terraform Prior experience in a More ❯

to lead and mentor a team of data engineers Experience in using techniques such as infrastructure as code and CI/CD Experience with graph-based data workflows using Apache Airflow Programming skills in one or more of the following: Python, Java, Scala, R and experience in writing reusable/efficient code to automate analysis and data processes Experience … in processing large volumes of data using parallelism techniques/tooling, such as Apache Spark Experience in processing structured and unstructured data into a form suitable for analysis and reporting with integration with a variety of data metric providers ranging from advertising, web analytics, and consumer devices Experience in basic Machine Learning techniques is a big plus Experience More ❯

/Financial Services domain is a plus. Preferred Qualifications Certifications in Pentaho, Big Data, or Cloud Platforms (AWS/GCP/Azure). Experience with Big Data technologies (Hadoop, Spark) and cloud data services. More ❯

Strong experience with Azure Data Lake Azure Synapse Azure SQL and Azure Storage Azure Data Factory ADF Expertise in building and managing complex data pipelines Databricks Handson experience with Spark based data processing notebooks and ML workflows Data Modeling Proficiency in conceptual logical and physical data modeling SQL Python Advanced skills for data manipulation and transformation Insurance Domain Knowledge More ❯

Strong experience with Azure Data Lake Azure Synapse Azure SQL and Azure Storage Azure Data Factory ADF Expertise in building and managing complex data pipelines Databricks Handson experience with Spark based data processing notebooks and ML workflows Data Modeling Proficiency in conceptual logical and physical data modeling SQL Python Advanced skills for data manipulation and transformation Insurance Domain Knowledge More ❯

for production data pipelines. Experience leveraging data modeling techniques and ability to articulate the trade-offs of different approaches. Experience with one or more data processing technologies (e.g. Flink, Spark, Polars, Dask, etc.) Experience with multiple data storage technologies (e.g. S3, RDBMS, NoSQL, Delta/Iceberg, Cassandra, Clickhouse, Kafka, etc.) and knowledge of their associated trade-offs. Experience with More ❯

with a strong track record of delivering impactful, production-ready solutions on Information Retrieval systems (e.g., recommender systems, search engines, vector-based search). Technical Expertise – Proficient in Python, Spark, and machine learning frameworks; strong understanding of statistical modeling and big data processing. Cloud Proficiency – Experience with AWS tools such as S3, SageMaker Studio, Glue Jobs, and Athena is More ❯

to explain commercial impact. Understanding of ML Ops vs DevOps and broader software engineering standards. Cloud experience (any platform). Previous mentoring experience. Nice to have: Snowflake or Databricks Spark, PySpark, Hadoop or similar big data tooling BI exposure (PowerBI, Tableau, etc.) Interview Process Video call - high-level overview and initial discussion In-person technical presentation - based on a More ❯

Python, Bash) and programming languages (Java). Hands-on experience with DevOps tools : GitLab, Ansible, Prometheus, Grafana, Nagios, Argo CD, Rancher, Harbour. Deep understanding of big data technologies : Hadoop, Spark, and NoSQL databases. Nice to Have Familiarity with agile methodologies (Scrum or Kanban). Strong problem-solving skills and a collaborative working style. Excellent communication skills , with the ability More ❯

a plus). Experience with model lifecycle management (MLOps), including monitoring, retraining, and model versioning. Ability to work across data infrastructure, from SQL to large-scale distributed data tools (Spark, etc.). Strong written and verbal communication skills, especially in cross-functional contexts. Bonus Experience (Nice to Have) Exposure to large language models (LLMs) or foundational model adaptation. Previous More ❯

a plus). Experience with model lifecycle management (MLOps), including monitoring, retraining, and model versioning. Ability to work across data infrastructure, from SQL to large-scale distributed data tools (Spark, etc.). Strong written and verbal communication skills, especially in cross-functional contexts. Bonus Experience (Nice to Have) Exposure to large language models (LLMs) or foundational model adaptation. Previous More ❯

cloud data platforms, Lakehouse architecture, and data engineering frameworks. Required Qualifications 6+ years of experience in data engineering 3+ years of hands-on experience with Databricks, Delta Lake, and Spark (PySpark preferred). Proven track record implementing Medallion Architecture (Bronze, Silver, Gold layers) in production environments. Strong knowledge of data modeling, ETL/ELT design, and data lakehouse concepts. … Proficiency in Python, SQL, and Spark optimization techniques. Experience working with cloud data platforms such as Azure Data Lake, AWS S3, or GCP BigQuery. Strong understanding of data quality frameworks, testing, and CI/CD pipelines for data workflows. Excellent communication skills and ability to collaborate across teams. Preferred Qualifications Experience with Databricks Unity Catalog and Delta Live Tables. More ❯