1 to 25 of 32 Delta Lake Jobs in the UK

solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … drive engineering best practices Skills & Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms ...

/ELT pipelines using PySpark, SQL and orchestration frameworks Automate and modernise legacy data processes with reusable patterns Design robust data models (Delta Lake, schema evolution, metadata, lineage) Ensure data security, governance, and compliance (access control, encryption, lifecycle) Provide technical leadership, mentoring and design reviews Collaborate with stakeholders … analysis, profiling and insight delivery Cloud & Platforms Azure: Data Factory, Synapse, Databricks, ADLS Gen2, Event Hub, Purview AWS: S3, Glue, Athena, Kinesis, Lambda, EMR, Lake Formation Databricks: Delta Lake, Unity Catalog, Structured Streaming, MLflow SQL Server stack (SSIS, SSRS) Engineering Practices CI/CD for data platforms ...

Databricks (Unity Catalog) Build scalable lakehouse and warehouse solutions using Databricks SQL and Fabric SQL endpoints Develop robust data pipelines with Databricks Workflows, Delta Live Tables (DLT), and Fabric Data Factory Define and implement cross-platform data strategies using Delta Lake to minimise duplication Establish data governance … Databricks (Unity Catalog) architecture Proven experience building modern lakehouse and data warehouse platforms Hands-on experience with: Databricks SQL & Fabric SQL Databricks Workflows & Delta Live Tables (DLT) Fabric Data Factory Deep knowledge of Delta Lake and cross-platform data strategies Solid experience in data governance, including lineage ...

data masking, encryption, lineage, access controls, and regulatory adherence (e.g., GDPR, ISO 27001). Design and optimize cloud-native storage and compute architectures using Delta Lake, ADLS Gen2, Synapse, Azure SQL, Cosmos DB, and NoSQL for petabyte-scale workloads. Implement event-driven and real-time streaming architectures using … Essential skills/knowledge/experience: (Up to 10) Azure Data Factory, Synapse Analytics, Azure Databricks ADLS, Azure Blob, Azure SQL DB, Cosmos DB, Delta Lake, Oracle DB Azure Event Hub, Azure Kafka, Azure IoT Hub, ADX Fabric including OneLake CI/CD using Azure DevOps, GitHub ...

timely, reliable and well-governed data across the organisation. You’ll: Design and develop scalable batch and streaming data pipelines using Databricks, Spark and Delta Lake Implement medallion architecture (bronze/silver/gold) to ensure data quality, lineage and efficient consumption Optimise Spark jobs and workflows … meet near real-time monitoring SLAs Build and maintain streaming pipelines using Structured Streaming and Delta Live Tables Ensure strong data governance , including schema enforcement, validation rules and ACID compliance Orchestrate and automate pipelines using Databricks Jobs, APIs or cloud-native schedulers Collaborate with data scientists, analysts and environmental ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

someone who wants to execute a defined plan. You will be working across two fronts simultaneously: building the core platform capabilities — the virtualized data lake, Common Data Layer, and data contract framework — that give WTW a unified, AI-ready data foundation; and embedding directly within product domains to deliver … access requirements. Background in financial services, insurance, or broking is a plus but not required. Technical Skills Deep expertise in Databricks, including Unity Catalog, Delta Lake, Delta Sharing, and the full Databricks data engineering and ML stack. Strong command of DBT for modular, testable, and well-documented ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

solutions Essential Skills Strong SQL and Python (PySpark preferred) Proven experience building cloud-based data platforms on Azure Hands-on experience with Azure Databricks, Delta Lake and Azure Data Lake Expertise designing ETL/ELT pipelines Experience with Power BI administration and data modelling Knowledge ...

modern cloud data engineering. Hands-on experience with Microsoft Fabric, including Lakehouse, Warehouse, Pipelines, Dataflows, Notebooks and Power BI. Good understanding of data lake, lakehouse and warehouse architecture. Experience building data ingestion and transformation pipelines from multiple source systems. Strong SQL skills and experience working with relational data models. … PySpark for data transformation and automation. Understanding of Power BI, semantic models, DAX and reporting requirements. Experience working with APIs, JSON, CSV, Parquet, Delta tables and common enterprise data formats. Knowledge of data governance, access control, data quality, metadata and lineage. Good documentation skills and the ability to produce ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

Engineering/Data Science, and Real-Time Analytics). Core Data Stack: High proficiency in advanced SQL , Python/PySpark, and data lakehouse architectures (Delta Lake). Cloud Infrastructure: Strong understanding of Microsoft Azure data services (Azure Data Factory, Azure Synapse, Azure SQL, Key Vault) as they relate ...

with OIDC federation) and/or Azure DevOps, supporting trunk-based development practices. Must Have Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/GOV.UK/Public Sector domain experience strongly preferred. Must ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must ...

data engineering.· 2+ years experience as data engineer on AI/ML project with Python· Strong experience using Databricks, including Jobs, Asset Bundles, Delta Lake, and MLflow, as well as Azure data engineering tools such as Azure Data Factory and Azure Data Lake Storage (ADLS).· Solid ...

Technical Lead, Lead Data Engineer, or Technical Delivery Lead position. Strong hands-on expertise with Python. Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and optimising large-scale ETL/ELT pipelines. Strong understanding of modern ...

building scalable data pipelines and data processing frameworks. Strong understanding of modern Lakehouse architecture principles. Experience working with: Unity Catalog/Databricks Workflows/Delta Live Tables/Databricks SQL. Experience integrating data from APIs and external data sources. Strong understanding of data modelling, data quality and performance optimisation … maintain high-performance data pipelines using Azure Databricks, PySpark, Spark SQL and Python. Contribute to the design and implementation of Lakehouse architecture using Delta Lake and Databricks best practices. Develop and manage data workflows using Databricks Workflows and Delta Live Tables. Utilise Unity Catalog to support effective ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

tool-specific implementation. 6-month initial contract What You'll Need Strong data engineering experience with solid software engineering fundamentals Hands-on Databricks experience (Delta Lake, Unity Catalog, Workflows, pipelines) Experience with lakehouse/medallion architectures Strong SQL + Python + Spark capability Familiarity with dbt, data quality ...

reporting. Core requirements: Strong experience delivering end-to-end data engineering solutions in production Deep hands-on expertise with Databricks (jobs, notebooks, workflows, Delta Lake) Experience designing and operating CI/CD pipelines for data platforms Strong Python & SQL skills for scalable data processing Solid understanding of lakehouse ...

Views, SLT, or equivalent SAP-native frameworks. Design and implement Medallion-aligned data pipelines (Bronze/Silver/Gold) in our cloud data lake or lakehouse (Azure/AWS/GCP/Databricks/Snowflake). Assess and remediate the current third-party implementation , identifying blockers and accelerating delivery. … strategy , including prioritisation of modules (FI/CO/MM/SD/PP etc.) required for December cutover. Build scalable ingestion pipelines using Delta Lake/Parquet and modern orchestration tools (ADF, dbx, Airflow, etc.). Collaborate with SAP functional teams to validate business logic, deltas ...

have the following skills and experience: Strong commercial experience in data architecture or data engineering within modern cloud environments. Deep expertise in Databricks, including Delta Lake, Unity Catalog, and SQL Warehousing. Advanced SQL skills with a focus on performance tuning and optimization. Experience migrating legacy data systems into ...

have the following skills and experience: Strong commercial experience in data architecture or data engineering within modern cloud environments. Deep expertise in Databricks, including Delta Lake, Unity Catalog, and SQL Warehousing. Advanced SQL skills with a focus on performance tuning and optimization. Experience migrating legacy data systems into ...

have the following skills and experience: Strong commercial experience in data architecture or data engineering within modern cloud environments. Deep expertise in Databricks, including Delta Lake, Unity Catalog, and SQL Warehousing. Advanced SQL skills with a focus on performance tuning and optimization. Experience migrating legacy data systems into ...