1 to 25 of 26 Delta Lake Jobs in the UK

solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … drive engineering best practices Skills & Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms ...

pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Proficiency in Python and SQL for data engineering and analytics TO BE CONSIDERED Please either ...

data masking, encryption, lineage, access controls, and regulatory adherence (e.g., GDPR, ISO 27001). Design and optimize cloud-native storage and compute architectures using Delta Lake, ADLS Gen2, Synapse, Azure SQL, Cosmos DB, and NoSQL for petabyte-scale workloads. Implement event-driven and real-time streaming architectures using … Essential skills/knowledge/experience: (Up to 10) Azure Data Factory, Synapse Analytics, Azure Databricks ADLS, Azure Blob, Azure SQL DB, Cosmos DB, Delta Lake, Oracle DB Azure Event Hub, Azure Kafka, Azure IoT Hub, ADX Fabric including OneLake CI/CD using Azure DevOps, GitHub ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

someone who wants to execute a defined plan. You will be working across two fronts simultaneously: building the core platform capabilities — the virtualized data lake, Common Data Layer, and data contract framework — that give WTW a unified, AI-ready data foundation; and embedding directly within product domains to deliver … access requirements. Background in financial services, insurance, or broking is a plus but not required. Technical Skills Deep expertise in Databricks, including Unity Catalog, Delta Lake, Delta Sharing, and the full Databricks data engineering and ML stack. Strong command of DBT for modular, testable, and well-documented ...

reliable 'single version of the truth' for reporting and analytics; - Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS); - Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … . Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing). - Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach. - Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

modern cloud data engineering. Hands-on experience with Microsoft Fabric, including Lakehouse, Warehouse, Pipelines, Dataflows, Notebooks and Power BI. Good understanding of data lake, lakehouse and warehouse architecture. Experience building data ingestion and transformation pipelines from multiple source systems. Strong SQL skills and experience working with relational data models. … PySpark for data transformation and automation. Understanding of Power BI, semantic models, DAX and reporting requirements. Experience working with APIs, JSON, CSV, Parquet, Delta tables and common enterprise data formats. Knowledge of data governance, access control, data quality, metadata and lineage. Good documentation skills and the ability to produce ...

Engineering/Data Science, and Real-Time Analytics). Core Data Stack: High proficiency in advanced SQL , Python/PySpark, and data lakehouse architectures (Delta Lake). Cloud Infrastructure: Strong understanding of Microsoft Azure data services (Azure Data Factory, Azure Synapse, Azure SQL, Key Vault) as they relate ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

with OIDC federation) and/or Azure DevOps, supporting trunk-based development practices. Must Have Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/GOV.UK/Public Sector domain experience strongly preferred. Must ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must ...

data engineering.· 2+ years experience as data engineer on AI/ML project with Python· Strong experience using Databricks, including Jobs, Asset Bundles, Delta Lake, and MLflow, as well as Azure data engineering tools such as Azure Data Factory and Azure Data Lake Storage (ADLS).· Solid ...

collaboratively within Agile delivery teams, ensuring adherence to engineering best practice. Required Skills & Experience Python/PySpark and SQL expertise. Proven Databricks experience, including Delta Lake and Unity Catalog. Understanding of Databricks Genie and its application within engineering workflows. Strong knowledge of Azure Data Factory and related Azure ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

Commercial teams to deliver scalable solutions What you bring Must-haves 8+ years in data engineering/data architecture Deep experience with Databricks (Lakehouse, Delta Lake, Unity Catalog) Expert SQL and distributed computing knowledge Strong experience with data modelling (star/snowflake schemas) Proven experience migrating legacy data ...

building scalable data pipelines and data processing frameworks. Strong understanding of modern Lakehouse architecture principles. Experience working with: Unity Catalog/Databricks Workflows/Delta Live Tables/Databricks SQL. Experience integrating data from APIs and external data sources. Strong understanding of data modelling, data quality and performance optimisation … maintain high-performance data pipelines using Azure Databricks, PySpark, Spark SQL and Python. Contribute to the design and implementation of Lakehouse architecture using Delta Lake and Databricks best practices. Develop and manage data workflows using Databricks Workflows and Delta Live Tables. Utilise Unity Catalog to support effective ...

Technical Lead, Lead Data Engineer, or Technical Delivery Lead position. Strong hands-on expertise with Python. Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and optimising large-scale ETL/ELT pipelines. Strong understanding of modern ...

reporting. Core requirements: Strong experience delivering end-to-end data engineering solutions in production Deep hands-on expertise with Databricks (jobs, notebooks, workflows, Delta Lake) Experience designing and operating CI/CD pipelines for data platforms Strong Python & SQL skills for scalable data processing Solid understanding of lakehouse ...

Views, SLT, or equivalent SAP-native frameworks. Design and implement Medallion-aligned data pipelines (Bronze/Silver/Gold) in our cloud data lake or lakehouse (Azure/AWS/GCP/Databricks/Snowflake). Assess and remediate the current third-party implementation , identifying blockers and accelerating delivery. … strategy , including prioritisation of modules (FI/CO/MM/SD/PP etc.) required for December cutover. Build scalable ingestion pipelines using Delta Lake/Parquet and modern orchestration tools (ADF, dbx, Airflow, etc.). Collaborate with SAP functional teams to validate business logic, deltas ...

have the following skills and experience: Strong commercial experience in data architecture or data engineering within modern cloud environments. Deep expertise in Databricks, including Delta Lake, Unity Catalog, and SQL Warehousing. Advanced SQL skills with a focus on performance tuning and optimization. Experience migrating legacy data systems into ...

have the following skills and experience: Strong commercial experience in data architecture or data engineering within modern cloud environments. Deep expertise in Databricks, including Delta Lake, Unity Catalog, and SQL Warehousing. Advanced SQL skills with a focus on performance tuning and optimization. Experience migrating legacy data systems into ...

have the following skills and experience: Strong commercial experience in data architecture or data engineering within modern cloud environments. Deep expertise in Databricks, including Delta Lake, Unity Catalog, and SQL Warehousing. Advanced SQL skills with a focus on performance tuning and optimization. Experience migrating legacy data systems into ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...