16 of 16 Permanent Apache Spark Jobs in Central London

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. • Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Team Leadership ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

Feature Store Containerisation and orchestration using Docker, Kubernetes or ECS Infrastructure as Code using Terraform or CloudFormation Large-scale data processing frameworks such as Spark or Dask Knowledge of data governance, compliance and regulated environments Experience delivering solutions within highly regulated industries such as government, finance or healthcare Security ...

quality programs, or analytics transformation Strong technical proficiency in data profiling, analysis, and data management using modern tools and environments, including Python, R, SQL, Spark, and cloud platforms Experience with data quality frameworks, including profiling, rule development, issue remediation, and the design of preventative controls Ability to translate complex ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

machine learning and data science frameworks. Hands-on experience with large language models, generative AI, prompt engineering, and RAG architectures. Experience with Databricks, MLflow, Spark, and cloud-based AI platforms (Azure preferred). Strong understanding of MLOps, software engineering best practices, and production deployment pipelines. Experience leading technical teams ...

Hands-on experience with ETL/ELT pipelines, data engineering, and large-scale data processing Experience with containerisation (Docker) and scalable data systems (e.g. Spark, Kafka) Nice-to-Have Experience with AWS SageMaker or similar managed ML platforms Background in safety-critical or regulated industries (aerospace, aviation, or similar ...

such as Government, Public Sector, Security, Justice, Defence, Healthcare or other similarly challenging environments . Experience with large-scale data technologies such as SQL, Spark or Hadoop would also be beneficial. About You You will be a technically credible leader who is still comfortable getting into the detail. ...

statistics, machine learning, and mathematical modelling. Experience designing evaluation frameworks for AI systems, including model performance and monitoring. Hands-on experience with Databricks, MLflow, Spark, or similar cloud-based data platforms. Knowledge of responsible AI, model governance, and explainability. Strong communication skills with the ability to explain complex technical ...

including Feast or AWS Feature Store. Docker, Kubernetes, or Amazon ECS. Infrastructure as Code using Terraform or CloudFormation. Big data processing technologies such as Spark or Dask. Experience working within regulated or highly secure environments. What's On Offer Work on meaningful AI programmes supporting critical national security missions. ...

delivery. Experience working across cross-functional teams including Architecture, Product, Engineering, and Business stakeholders. Data Engineering & Platforms Databricks Delta Lake Unity Catalog dbt Spark/PySpark Data pipeline development and optimisation Lakehouse architecture Data integration and ETL/ELT frameworks Data platform design and implementation Cloud Technologies … subject to confirmation) Start Date: ASAP/Short notice preferred Apply If you are an experienced Principal Data Engineer with expertise across Databricks, dbt, Spark, and AWS , and enjoy combining technical leadership with hands-on engineering delivery, we'd like to hear from you. Apply with your latest ...

market data normalisation. Experience handling time-series challenges such as timestamp precision, sequence gaps, duplicate and out-of-order events. Experience with distributed processing (Spark, Polars, Dask, Ray or similar). Experience with Kafka or similar streaming technologies. Familiarity with PostgreSQL, ClickHouse, Snowflake, Databricks, kdb+ or similar. Docker, Linux … market data engineering with proven experience building reliable, scalable data platforms. Our non-negotiable requirements are: Expert Python Strong SQL Scalable data pipeline development Apache Parquet Market data normalisation Automated data quality controls Deep tick and market data expertise Experience in MLOps is advantageous but not essential. We welcome ...

operate data workloads across cloud environments is preferred. Knowledge of Golang is a strong advantage. Hands‐on experience with data transformation tools such as Spark and DBT, with a focus on building well‐tested, maintainable data products is a strong advantage. Familiarity with Apache Iceberg table format ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you'll learn … scalability, reliability, and observability. Strong knowledge of the JVM and server-side Kotlin/Java programming. Solid understanding of the Hadoop ecosystem, including Spark, Hadoop, and Hive. Strong experience with cloud platforms and data infrastructure, especially AWS, including EMR, S3, and Glue. Experience working in Agile environments and contributing ...