1 to 25 of 141 Delta Lake Jobs in the UK

/CD, Git and automated deployment Monitoring, logging and operational alerting Data-quality and reconciliation controls API/file/SFTP integration patterns Delta Lake/Delta MERGE Databricks Asset Bundles Azure DevOps MLOps/model deployment experience SFMC or Comarch integration Customer decisioning/personalisation platforms … Nice To Have Delta Lake/Delta MERGE Databricks Asset Bundles Azure DevOps MLOps/model deployment experience SFMC or Comarch integration Customer decisioning/personalisation platforms MLOps & Integration Data Engineer Role Overview BNA Production-Readiness, MLOps & Integration Engineer: Automated end-to-end BNA orchestration Reliable retry ...

Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus on common IDs, Golden Records … survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity Catalog, data lineage ...

Experience: 610+ Years Key Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus … survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity Catalog, data lineage ...

build of large-scale data platforms using Azure Databricks and modern cloud technologiesImplement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principlesEnhance performance, manage cloud costs and ensure platform reliability for structured streaming workloadsDefine data governance, security and quality standards to maintain … including advanced optimization, data skew mitigation and query tuningStrong programming skills in Python with knowledge of modern software design principlesPractical experience with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake ...

scalable data pipelines using Dataflow Gen2 and Fabric Notebooks • Develop and maintain Lakehouse solutions using Medallion architecture • Work with SQL, Python, Spark SQL and Delta Lake • Integrate data from APIs, databases, CRM, ERP and third party platforms • Support both batch and real time data processing • Develop semantic models … Notebooks and Lakehouse architecture • Strong SQL and Python skills • Experience with Spark and Spark SQL • Proven experience designing scalable ETL and ELT pipelines • Data Lake and Delta Lake experience • Experience integrating data from multiple structured and unstructured sources • Good understanding of CI/CD, DevOps and modern ...

raise risks early As part of your duties, you will be responsible for: Architecture and modelling Design the lakehouse architecture on Azure Databricks using Delta Lake and the Medallion pattern (bronze, silver and gold layers), organising data by quality and purpose to keep the platform scalable and maintainable. … reporting and self‐service analytics. Select and design for the right storage and exchange formats for each workload, including row‐based and columnar storage, Delta and Iceberg table formats, and file and message formats (e.g. Parquet, CSV, Excel, XML, JSON, Protobuf). Optimise platform performance and cost, including partitioning ...

dense/hybrid retrieval, reranking, and semantic search. Data Platform Engineering: Build scalable, streaming, and batch data processing pipelines across modern cloud lakehouse architectures (Delta Lake, streaming architectures, and unified governance layers). Systematic Evaluation & Benchmarking: Establish continuous model evaluation, reproducibility checks, tracing, and benchmarking frameworks (MLflow … Agentic Frameworks: Proven delivery with LangGraph, AutoGen, CrewAI, smolagents, or Model Context Protocol (MCP) integrations. Lakehouse & Data Engineering: Strong background with enterprise Databricks environments (Delta Lake, Unity Catalog, Delta Live Tables/DLT, Spark Structured Streaming, or Autoloader). Cloud & Model Ecosystem: Broad exposure to Azure cloud ...

dense/hybrid retrieval, reranking, and semantic search. Data Platform Engineering: Build scalable, streaming, and batch data processing pipelines across modern cloud lakehouse architectures (Delta Lake, streaming architectures, and unified governance layers). Systematic Evaluation & Benchmarking: Establish continuous model evaluation, reproducibility checks, tracing, and benchmarking frameworks (MLflow … Agentic Frameworks: Proven delivery with LangGraph, AutoGen, CrewAI, smolagents, or Model Context Protocol (MCP) integrations. Lakehouse & Data Engineering: Strong background with enterprise Databricks environments (Delta Lake, Unity Catalog, Delta Live Tables/DLT, Spark Structured Streaming, or Autoloader). Cloud & Model Ecosystem: Broad exposure to Azure cloud ...

develop and manage STACK's enterprise data platform. This is a production-focused engineering role requiring practical expertise in Databricks, PySpark, Python, SQL and Delta Lake. The role will design reliable pipelines integrating Finance, Delivery, Operations, HR and PMO systems, establish governed and high-quality datasets, and support secure … production-grade ETL/ELT pipelines using Databricks, PySpark, Python and SQL, integrating APIs, databases, SaaS applications, files and spreadsheets. Develop and optimize scalable Delta Lake data models and lakehouse architecture for reporting, analytics and future AI use cases. Build, schedule, monitor and troubleshoot data processing jobs using ...

helping clients make appropriate architecture and technology decisions. Databricks & Modern Data Platforms Design enterprise Databricks and Lakehouse solutions incorporating technologies and patterns such as Delta Lake, medallion architecture, Unity Catalog and Databricks SQL. Apply strong knowledge of modern cloud data ecosystems, including Azure and/or AWS, with … delivering enterprise cloud data architectures within complex client or business environments. Deep expertise in Databricks and modern Lakehouse architecture, including practical knowledge of Delta Lake and Unity Catalog. Strong experience with Microsoft Azure and/or AWS and modern enterprise data platforms, including Databricks, Microsoft Fabric or another ...

APIs and file-based sources. Build and optimise SQL and PySpark transformations across Bronze, Silver and Gold data layers. Work with technologies including Databricks, Delta Lake, Lakeflow, dbt and Unity Catalog. Implement automated data quality checks, monitoring and alerting to ensure data accuracy, completeness and reliability. Manage … window functions and data modelling. Good Python programming skills, particularly PySpark, for data transformation, API integrations and automation. Hands-on experience with Databricks, including Delta Lake, Spark and Unity Catalog, or a comparable cloud data platform such as Snowflake or AWS Redshift. Experience working with data ingestion techniques ...

GitHub Actions and Terraform setup for deployments Implement, run and monitor dbt tests to ensure data quality and consistent dimensional models Extend and maintain Delta Lake fact and dimension tables for analytics use cases, following a bronze/silver/gold-style data architecture Work closely with analysts … they bring clear value, including integrating AI tools for analysis and AI coding agents into day-to-day workflows Tech stack Azure Databricks (Spark, Delta Lake) SQL Server, Confluent Kafka Scala, SQL, dbt Git, GitHub Actions, Terraform Sigma (BI) Must-have experience Commercial experience as a Data Engineer ...

data from internal systems and third-party platforms including Salesforce, Sopra Banking Services, CACI and Experian. Engineer Modern Data Architectures - Design Lakehouse, Warehouse and Delta Lake solutions using medallion architectures to support enterprise analytics and reporting. Deliver Analytics-Ready Data Assets - Build trusted, well-modelled datasets and semantic … successfully built, enhanced or operated cloud‐based data platforms, ideally within Azure environments. Data Architecture Knowledge - You understand data modelling, ELT/ETL design, Delta Lake principles, data quality frameworks and modern engineering best practices. Cloud & DevOps Capability - You have experience with Azure services, automation, CI/ ...

Project description Designed and implemented a scalable enterprise Lakehouse platform on Azure using Azure Data Factory, Delta Lake, ADLS Gen2, and Databricks. Migrated legacy ETL processes to cloud-native ELT pipelines supporting both batch and near real-time workloads. Established reusable data frameworks, metadata-driven orchestration, data quality … validation, and automated CI/CD pipelines. Responsibilities Designed and implemented scalable data pipelines using Azure Data Factory, Databricks, Delta Lake, Python, and SQL.Architected Lakehouse solutions supporting enterprise analytics, AI/ML, and Generative AI workloads.Developed structured and unstructured data pipelines for RAG, vector search, and AI-powered ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

frameworks and model benchmarking methodologiesAdvanced programming skills in Python and SQL with strong knowledge of data modeling and ETL/ELT processesSolid understanding of Delta Lake, Delta Live Tables, Unity Catalog, Spark Structured Streaming and large-scale data processingExperience with Azure Data Factory (ADF), Qlik Replicate ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

timely, reliable and well-governed data across the organisation. You'll: Design and develop scalable batch and streaming data pipelines using Databricks, Spark and Delta Lake Optimise Spark jobs and workflows to meet near real-time monitoring SLAs Build and maintain streaming pipelines using Structured Streaming and Delta … technical leader and mentor , raising standards across the data engineering community Essential skills & experience Deep expertise in Databricks (Spark/PySpark, Delta Lake, Unity Catalog, Workflows) Strong experience building ETL/ELT pipelines , both batch and streaming Advanced Python and SQL skills Hands-on experience with cloud platforms ...

Architect and lead the end-to-end delivery of Databricks-based data platforms across Azure environments. Design secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate. Establish and promote engineering best practices across … approved AI tools and automation to enhance delivery and collaboration. What We're Looking For Proven expertise with Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments. Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data ...

supporting colleagues. What you'll do ·Build and optimise pipelines using Azure Databricks, PySpark, SQL, ADF and ADLS Gen2. ·Develop governed lakehouse solutions using Delta Lake, medallion architecture and Unity Catalog. ·Own Power BI semantic models, DAX measures and row-level security, ensuring reporting is accurate, secure …/CD. ·Mentor colleagues and share good practice across the team. What you'll bring ·Strong production experience with Databricks, PySpark, Delta Lake and Unity Catalog. ·Hands-on Power BI expertise, including data modelling, DAX, RLS and performance optimisation. ·Strong SQL skills and experience with Azure data services. ...

Required Skills & Experience 3+ years of experience within Data Engineering, Analytics Engineering, or Data Analytics. Strong hands-on experience with: Databricks SQL Python PySpark Delta Lake Experience designing ETL/ELT pipelines and data transformation frameworks. Strong understanding of data modelling, performance optimisation, and data quality practices. Experience … cutting-edge AI-assisted development practices. Challenging, business-critical projects with significant impact. Collaborative and innovative working environment. Keywords: Databricks, Python, SQL, PySpark, Delta Lake, Data Engineer, Analytics Engineer, Data Analytics, ETL, ELT, AI Engineer, Generative AI, GitHub Copilot, ChatGPT, Claude Code, Cursor, Prompt Engineering, Context Engineering, Data ...

schema integrity. Key Responsibilities Source-to-Target Reconciliation: Design, build, and execute automated validation queries to compare legacy eFront extracts against target Databricks Delta Lake tables (Bronze, Silver, Gold). Business Logic & Metrics Testing: Verify complex private equity calculation logic (cash flows, commitments, valuations, NAV, and return metrics … Engineer, Data Quality Engineer, or Data Test Automation Engineer on enterprise cloud migrations. Hands-on testing and reconciliation experience within Azure Databricks/Delta Lake environments. Familiarity with private markets, fund accounting, or private equity financial structures (NAV, cash flows, capital calls, distributions). #J-18808-Ljbffr ...

shared data assets Technical Leadership & Standards Serve as a senior technical authority for data product architecture across the program Define architectural best practices for Delta Lake design, metadata management, product lifecycle governance, and data quality controls Provide architecture assurance and design reviews for data products developed by internal … data contracts, service models, and consumer engagement frameworks Excellent stakeholder management, communication, and consulting skills Technical Skills Deep expertise in Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, and Databricks SQL Strong understanding of data modelling, schema design, metadata management, and lineage tracking Experience implementing self-service data ...

managing and reporting control issues with transparency. Skills & Experience Essential Proven in-depth commercial experience in the following key areas: Databricks – Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning. Apache Spark & Scala – High-performance distributed data processing and optimisation. Cloud Data Engineering (AWS) – Cloud-native data platforms … core business functions Demonstrated leadership, project management, and development skills Relationship and consensus building skills Preferred Kafka/Event-Driven Architecture Apache Iceberg/Delta Lake Python AI/ML & LLM integration Financial Services/Capital Markets Education Bachelor’s degree/University degree or equivalent experience Master ...

risks, exceptions, and technical dependencies raised through governance processes Enterprise & Solution Architecture Provide architecture assurance for Databricks Lakehouse implementations, including data ingestion, medallion architecture, Delta Lake, Unity Catalog, and compute strategies Validate integration patterns across Databricks, cloud platforms, enterprise applications, and data ecosystems Define and maintain reference architectures … delivery requirements Exceptional stakeholder engagement and influencing skills across technical and business audiences Technical Skills Strong hands-on knowledge of Databricks including Lakehouse Architecture, Delta Lake, Unity Catalog, and platform governance Experience with cloud platforms supporting Databricks environments, including AWS and/or Azure Solid understanding of enterprise ...