1 to 25 of 311 Apache Spark Jobs in London

design, develop, deploy, and optimize scalable data solutions on Microsoft Azure. The ideal candidate will bring strong hands-on expertise in Azure Databricks, Apache Spark, PySpark, Python, and SQL, with proven experience building production-grade data pipelines, implementing governance through Unity Catalog, and automating deployments using GitLab-based …/CD and Databricks Asset Bundles. Key Responsibilities Design, build, test, and maintain scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake. Develop reusable ETL and ELT frameworks for data ingestion, transformation, validation, and publishing across lakehouse layers. Design ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Build, lead ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities.Team Leadership & Delivery ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. • Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Team Leadership ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

Solution Engineering, or related disciplines within enterprise environments4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse PlatformStrong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practicesExperience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … artificial intelligence solutions on the Databricks Lakehouse Platform.Design and implement scalable data and analytics solutions leveraging Databricks technologiesBuild, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta LakeDevelop governed datasets and data products that support reporting, analytics, and artificial intelligence initiativesPartner with business stakeholders ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

frameworks, tools and applications that make that data available to other teams and systems.Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

platforms.Solid Cloud and DevOps knowledge, including Linux/Unix, OpenShift/Kubernetes, and Ansible.Experience building and supporting reactive, event-sourced systems.Hands-on experience with Apache Spark and Apache Flink.Strong understanding of Apache HTTP, Envoy, and REST API design.Experience working with Protobuf, Avro, and Parquet data formats.Knowledge ...

DevOps knowledge, including Linux/Unix, OpenShift/Kubernetes, and Ansible. Experience building and supporting reactive, event-sourced systems. Hands-on experience with Apache Spark and Apache Flink. Strong understanding of Apache HTTP, Envoy, and REST API design. Experience working with Protobuf, Avro, and Parquet data ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production.Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline … lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

ingestion, incremental loading, and change data capture. Design and implement layered data models and transformation logic (raw, conformed, aggregated) using tools such as dbt, Spark, and cloud‐native processing services, ensuring data is clean, consistent, and ready for downstream consumption. Develop modular, reusable data transformation components and functions that … tooling, or cloud cost management across at least one major cloud provider (Azure, AWS, or GCP). Technical Skills Data engineering: strong proficiency with Apache Spark, dbt, Airflow/Azure Data Factory or equivalent orchestration, Data Lake/Apache Iceberg, and advanced SQL for large‐scale transformations. ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands‐on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud‐native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

semi-structured, and unstructured dataHands-on expertise in containerization and orchestration using Docker and Kubernetes for scalable production systemsFamiliarity with distributed computing frameworks (e.g., Spark, Dask), cloud platforms (e.g., AWS, Azure, GCP), and analytics libraries (e.g., pandas, numpy, matplotlib)Exposure to DevOps, DataOps, and MLOps concepts and best practicesExperience ...

building, and operating ETL/ELT pipelines and large-scale data architectures.Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential).Ability ...