1 to 25 of 82 Delta Lake Jobs in the UK

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

drives real-time data, analytics, and reporting across the Tote.You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast, trusted data at their fingertips. … governed data solutions under Unity Catalogue.What you’ll be doing:You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop robust Bronze, Silver ...

deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products.Job responsibilitiesDesign and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake, and orchestrators (e.g., Workflows, Airflow, ADF).Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost … pipelines on Databricks and/or Apache Spark in production.Strong coding skills in Python (PySpark) and SQL (Scala a plus).Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM).Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure Data ...

Data Engineer, you’ll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks.Responsibilities:Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue.Data Transformation: Develop, maintain, and optimize … transformation pipelines within our Delta Lake on Databricks.Data Modelling: Design and implement efficient and reliable data models to store and retrieve data.Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions.Data Quality: Ensure the reliability, integrity, and timeliness of data ...

scalability, security, and governance. Act as Databricks champion: define standards, reference architectures, accelerators, and reusable assets. Lead architectural design for Databricks (workspaces, clusters, jobs, Delta Lake, Unity Catalogue, MLflow) and integrations with cloud services. Partner with pre-sales teams to shape propositions, run discovery, produce high-quality proposals … encryption, auditing, lineage). Essential Requirements Significant Databricks experience required, including production-scale deployments and optimisation. Strong hands-on capability with Spark/SQL, Delta Lake, orchestration, and data engineering patterns. Prior experience within a consulting practice with exposure to both pre-sales and delivery leading ...

pipelines that transform scientific datasets into analysis-ready formats aligned with FAIR data principles.Lead Databricks integration within Product Safety, driving best practices across Delta Lake, Unity Catalog, and data governance.Deliver strategic data product prototypes and partner with the Safety and Regulatory Data Domain Lead to transition these into … strong data engineering skills with the ability to work fluently across scientific and technical domains.Essential skills and experience:Proven data engineering experience with Databricks (Delta Lake, Unity Catalog, Apache Spark) in cloud-based data platforms and environments (AWS).Strong programming skills in Python and SQL, with experience ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. Cloud platform ...

practices.What You'll DoArchitect and lead the end-to-end delivery of Databricks-based data platforms across Azure environments.Design secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate.Establish and promote engineering best practices across … approved AI tools and automation to enhance delivery and collaboration.What We're Looking ForProven expertise with the Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments.Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data platforms.Extensive experience ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. ...

data engineering to improve data quality, enforce security standards, and maintain lineage, metadata, and documentationImplement and optimise data pipelines and analytics models using Spark, Delta Lake, and cloud-native toolingCoach and support analytics engineers and analysts through code reviews, mentoring, and best practice adoptionTranslate stakeholder requirements into clear … stack (Power BI, SQL Server/SSRS, Excel)Expert in data modelling (dimensional and lakehouse architectures) and enterprise dataset designAdvanced Databricks and Spark engineering (Delta Lake, performance tuning, orchestration, schema evolution)Strong understanding of cloud data platforms (Azure or AWS) and secure, scalable data architecturesExperience implementing data quality ...

scalable ETL/ELT data pipelines. Microsoft Fabric, including Data Factory, Dataflow Gen2 and Fabric Notebooks. Databricks Workflows and Lakehouse technologies. PySpark, SQL and Delta Lake environments. Data ingestion, transformation and orchestration processes. Data governance, security and quality initiatives. Enterprise-wide analytics and reporting solutions, including Power … data transformation best practices. Experience building and supporting production-grade data pipelines. Hands-on experience with Microsoft Fabric. Experience with Databricks and Delta Lake technologies. Understanding of Lakehouse architecture and modern data platforms. Experience delivering data solutions that support analytics and business reporting. Interested? This is a rare ...

deliver actionable, enterprise-grade business insights.Key ResponsibilitiesData Architecture & CDL Governance: Design, implement, and govern the enterprise Common Data Layer (CDL) on Azure Databricks using Delta Lake and L0–L1–L2 architectures. Establish data quality frameworks, schema evolution standards, and deduplication logic.Pipeline & Integration Engineering: Oversee the creation of scalable … connection of Azure Databricks data assets into robust, secure, and intuitive Power BI dashboards.Required Experience & SkillsTechnical ExpertiseCloud Data Platforms: Advanced knowledge of Azure Databricks, Delta Lake, Azure Data Factory (ADF), and SQL querying.SAP Ecosystem: Deep data discovery skills within SAP S/4HANA, SAP BW/HANA ...

Medallion Architecture, and ensuring our Databricks platform is secure, governed, and production ready.Key Responsibilities: Design, build, and optimise scalable data pipelines using Databricks, Delta Live Tables, Synapse, and Delta Lake.Develop ingestion, transformation, and orchestration workflows across diverse sources (APIs, SQL, SFTP, SaaS platforms).Implement strong data quality, lineage … across pipelines and clusters.Contribute to standards for versioning, DevOps, CI/CD, and testing for Databricks notebooks, workflows, and SQL assets.Support the rollout of Delta Sharing, Unity Catalog governance, and Iceberg/Delta interoperability where required.Partner with analysts and data scientists to ensure that high-quality, well-modelled ...

including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices (testing ...

communicate complex concepts in a clear, concise manner• Languages: Terraform, Python, SQL, PowerShell.• CI/CD: Azure DevOps pipelines for data platform deployments.• Databricks: Delta Lake, Unity Catalog, Workflows, Notebooks, Clusters, MLflow.• Azure: ADLS Gen2, Azure Data Factory, Event Hubs/Kafka, Key Vault, Functions, Synapse. AI Foundry … Service Plan Deployment and Configuration, Integration with Azure bot Services• MS Fabric: Tenant Management, Workspace Private Links, Deployment, Capacity Management, Workspace Management, One Lake, Data Mirroring, Power BI• AI Foundry : Model management, Hosted Agents Deployment and Management , Project Configuration (Connections and Governance ) and lifecycle management.#LI-PM1Inclusion & DiversityWe value diversity ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

Strong Azure Data Factory (ADF) and Azure Databricks experience. Experience building ETL/ELT pipelines within Azure Lakehouse environments. Knowledge of Medallion Architecture and Delta Lake. Strong SQL and Python skills. Experience with Power BI datasets and data modelling. Familiarity with Azure DevOps, Git, CI/CD and Infrastructure ...

systematic approach to diagnosing and resolving issuesHighly desirable skillsExperience with streaming frameworks such as Flink, Kafka Streams and Kafka ConnectKnowledge of modern data lake technologies including Delta Lake, Iceberg and Glue Data CatalogExposure to DataOps practices and collaboration with Data Engineering teamsFamiliarity with GPU-accelerated analytics using ...

platforms (preferably AWS); production experience is not required.A collaborative mindset and willingness to learn and communicate openly within the team.Nice to haveExposure to data lake concepts or formats such as Delta Lake or Iceberg.Any experience with search, graph, or large-scale analytical databases.Interest in AI, machine learning ...

files, and event streams Supporting master data management, data quality rules, and golden record definition Working hands-on with ADLS, Azure SQL, Event Hubs, Delta Lake, and OneLake Designing integration patterns and REST APIs to support near real-time data flows Contributing to infrastructure-as-code and environment ...

remaining actively involved in hands-on delivery. Experience working across cross-functional teams including Architecture, Product, Engineering, and Business stakeholders. Data Engineering & Platforms Databricks Delta Lake Unity Catalog dbt Spark/PySpark Data pipeline development and optimisation Lakehouse architecture Data integration and ETL/ELT frameworks Data platform ...

such as Snowflake's Snowpark, Dynamic Tables, Streams & Tasks, and Snowflake Cortex, as well as Databricks components such as Lakeflow (Connect, Pipelines, and Jobs), Delta Lake, and Unity Catalog. Strong data engineering fundamentals: ETL/ELT pipeline design and implementation, data orchestration and workflow automation, batch and streaming ...

models (conceptual, logical, and physical), including lakehouse zone strategies and schema evolution approaches.Champion Databricks as the core data platform, leveraging lakehouse architecture, Unity Catalog, Delta Live Tables, Structured Streaming, and MLflow.Drive a consistent multi-cloud Databricks strategy, enabling scalable and interoperable data platforms across environments.Optimise performance across pipelines, compute … multi-environment promotion.Embed robust data governance, security, and lifecycle controls, ensuring compliance through classification, encryption, access management, and lineage frameworks (e.g., Unity Catalog, Purview, Lake Formation).Act as a senior SME for troubleshooting complex issues, leading root-cause analysis and long-term remediation of data quality and platform challenges.Lead ...

pipeline development, and large-scale data processing.Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).Strong understanding of data quality, security … Step Functions, etc.).Databricks or AWS certifications.Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats.J.P. Morgan is a global leader in financial services, providing strategic advice and products to the world’s most prominent corporations ...

PySpark GitLab/Azure DevOps pipelines Experience validating large-scale datasets and data transformations Knowledge of data warehousing concepts and dimensional modeling Understanding of Delta Lake, Parquet, and Lakehouse architectures Experience with CI/CD and automated deployment validation Familiarity with Agile/Scrum delivery methodologies Microsoft Fabric ...