1 to 25 of 429 Apache Spark Jobs in the UK

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

data processing. In this role, you will architect and maintain production-grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high-performance solutions. You will collaborate with cross-functional partners across Finance, Accounting, and Analytics to translate complex … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine-tune complex queries ...

design, implement, and optimize enterprise-scale data platforms using the Databricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions. Key Responsibilities Design and architect Databricks Lakehouse solutions for large-scaleenterprise data … Databricks implementation, including architecture, development, deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Build, lead ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities.Team Leadership & Delivery ...

advanced data analysis, modelling, and visualization to identify patterns and insights.Ensure data quality, reliability, and security across systems. Work with distributed computing frameworks (e.g., Spark, Hive, Hadoop) to process and analyse large datasets. Stay up to date with the latest trends and technologies in big data, analytics, and cloud … programming skills in languages such as Python, Java, or Scala.Hands-on experience with Databricks, SQL, and data modelling. Proficiency with big data tools (e.g., Apache Spark, Apache Kafka, Presto, etc.) and cloud services (e.g., Azure, AWS, GCP).Familiarity with data visualization tools and business intelligence (e.g., Power ...

data analysis, modelling, and visualization to identify patterns and insights. Ensure data quality, reliability, and security across systems. Work with distributed computing frameworks (e.g., Spark, Hive, Hadoop) to process and analyse large datasets. Stay up to date with the latest trends and technologies in big data, analytics, and cloud … skills in languages such as Python, Java, or Scala. Hands-on experience with Databricks, SQL, and data modelling. Proficiency with big data tools (e.g., Apache Spark, Apache Kafka, Presto, etc.) and cloud services (e.g., Azure, AWS, GCP). Familiarity with data visualization tools and business intelligence (e.g. ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

Databricks-based Lakehouse that drives real-time data, analytics, and reporting across the Tote.You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast … deliver scalable, secure, and governed data solutions under Unity Catalogue.What you’ll be doing:You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop ...

databases utilised in modern AI applications. Strong Python development skills. Data & Infrastructure: Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure: Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases: PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Ecosystem: Pandas Polars ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

Solution Engineering, or related disciplines within enterprise environments4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse PlatformStrong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practicesExperience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … artificial intelligence solutions on the Databricks Lakehouse Platform.Design and implement scalable data and analytics solutions leveraging Databricks technologiesBuild, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta LakeDevelop governed datasets and data products that support reporting, analytics, and artificial intelligence initiativesPartner with business stakeholders ...

related disciplines within enterprise environments 4+ years of hands‐on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain). Hands-on experience working with Distributed Spark based systems Experience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external … clients across a variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred [Preferred] Experience working with Apache Spark to process large-scale distributed datasets Communication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences. Core ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

frameworks, tools and applications that make that data available to other teams and systems.Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

platforms.Solid Cloud and DevOps knowledge, including Linux/Unix, OpenShift/Kubernetes, and Ansible.Experience building and supporting reactive, event-sourced systems.Hands-on experience with Apache Spark and Apache Flink.Strong understanding of Apache HTTP, Envoy, and REST API design.Experience working with Protobuf, Avro, and Parquet data formats.Knowledge ...

DevOps knowledge, including Linux/Unix, OpenShift/Kubernetes, and Ansible. Experience building and supporting reactive, event-sourced systems. Hands-on experience with Apache Spark and Apache Flink. Strong understanding of Apache HTTP, Envoy, and REST API design. Experience working with Protobuf, Avro, and Parquet data ...

DevOps knowledge, including Linux/Unix, OpenShift/Kubernetes, and Ansible. Experience building and supporting reactive, event‐sourced systems. Hands‐on experience with Apache Spark and Apache Flink. Strong understanding of Apache HTTP, Envoy, and REST API design. Experience working with Protobuf, Avro, and Parquet data ...

data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc Experience developing end‐to‐end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam. Expert understanding of when to use a range of data storage technologies including relational/non‐relational, document ...

ready data processing solutions using BigQuery, Pub/Sub, Dataflow and DataprocExperience developing end-to-end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam.Expert understanding of when to use a range of data storage technologies including relational/non-relational, document ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production.Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline … lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices ...

with Red Hat OpenShift and Ceph storage platforms. Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark). Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training ...