1 to 25 of 82 Delta Lake Jobs in the UK

feeds honest, and ownership of the data warehouse and the data sync between our ERP and the wider estate. Our stack is Python, PySpark, Delta Lake and Databricks, with T‐SQL, Kimball dimensional modelling and Power BI on top, working closely with Microsoft Dynamics 365 Finance and Operations … stakeholders trust, from the models underneath through to well built, performant Power BI reports. Design and build data pipelines in Python, PySpark and Delta Lake, and write the T‐SQL behind our models and reconciliations. Keep an eye on cost, performance and reliability across the platform, and keep ...

First options availableThe RoleAs a Data Engineer, you will focus on designing, developing, and optimising data ingestion, transformation, and serving layers across Azure Databricks, Delta Lake, and Databricks SQL. Your work will enable reliable data products and insights for stakeholders, and you’ll collaborate with data architects, analysts … needs.You WillBuild and maintain metadata-driven pipelines using Azure Databricks to ingest and transform batch and streaming data, ensuring data quality and scalable processing.Develop Delta Lake layers following bronze/silver/gold patterns, and model data optimised for BI and analytics, including Power BI integration through Databricks ...

First options availableThe RoleAs a Data Engineer, you will focus on designing, developing, and optimising data ingestion, transformation, and serving layers across Azure Databricks, Delta Lake, and Databricks SQL. Your work will enable reliable data products and insights for stakeholders, and you'll collaborate with data architects, analysts … WillBuild and maintain metadata-driven pipelines using Azure Databricks to ingest and transform batch and streaming data, ensuring data quality and scalable processing. Develop Delta Lake layers following bronze/silver/gold patterns, and model data optimised for BI and analytics, including Power BI integration through Databricks ...

Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus on common IDs, Golden Records … survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity Catalog, data lineage ...

Experience: 610+ Years Key Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus … survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity Catalog, data lineage ...

build of large-scale data platforms using Azure Databricks and modern cloud technologiesImplement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principlesEnhance performance, manage cloud costs and ensure platform reliability for structured streaming workloadsDefine data governance, security and quality standards to maintain … including advanced optimization, data skew mitigation and query tuningStrong programming skills in Python with knowledge of modern software design principlesPractical experience with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake ...

scalable data pipelines using Dataflow Gen2 and Fabric Notebooks • Develop and maintain Lakehouse solutions using Medallion architecture • Work with SQL, Python, Spark SQL and Delta Lake • Integrate data from APIs, databases, CRM, ERP and third party platforms • Support both batch and real time data processing • Develop semantic models … Notebooks and Lakehouse architecture • Strong SQL and Python skills • Experience with Spark and Spark SQL • Proven experience designing scalable ETL and ELT pipelines • Data Lake and Delta Lake experience • Experience integrating data from multiple structured and unstructured sources • Good understanding of CI/CD, DevOps and modern ...

Server and Databricks. Support Lakeflow pipelines, production jobs and data quality processes. Monitor, troubleshoot and optimise data workflows and platform performance. Work with Delta Lake, Unity Catalog and modern data lake architectures. Integrate Azure services including ADLS, Data Factory, Key Vault and REST APIs. Support CI/… enterprise environment. Excellent SQL Server/T-SQL and data engineering skills. Experience building and optimising large-scale data pipelines. Knowledge of Spark, Delta Lake, Unity Catalog and Azure data services. Exposure to Git, CI/CD and infrastructure automation tools. Strong communication skills and a proactive approach ...

data solutions, from conceptual and logical models through to physical implementation Building and maintaining scalable pipelines and lakehouse solutions using Azure Databricks, Spark and Delta Lake Designing and optimising Snowflake environments for performance, scalability and cost Supporting batch, streaming and real-time data processing, including high-volume transactional … environments Hands-on, current engineering skills: you still write the code, not just the diagrams Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake and Lakehouse architecture Strong Snowflake architecture and optimisation experience Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics and/or Microsoft ...

Data Engineer, you’ll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks.Responsibilities:Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue.Data Transformation: Develop, maintain, and optimize … transformation pipelines within our Delta Lake on Databricks.Data Modelling: Design and implement efficient and reliable data models to store and retrieve data.Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions.Data Quality: Ensure the reliability, integrity, and timeliness of data ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

frameworks and model benchmarking methodologiesAdvanced programming skills in Python and SQL with strong knowledge of data modeling and ETL/ELT processesSolid understanding of Delta Lake, Delta Live Tables, Unity Catalog, Spark Structured Streaming and large-scale data processingExperience with Azure Data Factory (ADF), Qlik Replicate ...

best practices across the enterprise data ecosystem. The role leads the design and delivery of scalable, secure, high-performing data products using Databricks, Delta Lake, Unity Catalog and Azure Data Services The role provides technical leadership, ensuring solutions are robust, cost-effective, highly available, and aligned to business … development. To ensure data dictionaries and models are robust and maintained on a regular basis. Technical Skill Expert-level Databricks experience Advanced PySpark development Delta Lake architecture and optimisation Databricks SQL Data warehouse Dimensional modelling Data Lake Databricks Genie Python development Key Skills, Experience and Knowledge Extensive ...

cost and usage data platform on Databricks, ingesting Azure Cost Management exports, reservation and Savings Plan data, resource inventory and tagging metadata into Delta Lake and modelling it into trusted datasets. Build and own Power BI semantic models behind cost reporting, including DAX measures, composite and DirectQuery designs … rather than simply creating reports. Nice to HaveFinOps or cloud-cost experience, including Azure Cost Management exports, FOCUS, reservations or Savings Plans. Databricks and Delta Lake. KQL.TMDL and XMLA.PowerShell. Experience with cloud cost optimisation, allocation/showback or cost attribution. What We're UsingPython — production services, data pipelines, automation ...

supporting colleagues. What you'll do ·Build and optimise pipelines using Azure Databricks, PySpark, SQL, ADF and ADLS Gen2. ·Develop governed lakehouse solutions using Delta Lake, medallion architecture and Unity Catalog. ·Own Power BI semantic models, DAX measures and row-level security, ensuring reporting is accurate, secure …/CD. ·Mentor colleagues and share good practice across the team. What you'll bring ·Strong production experience with Databricks, PySpark, Delta Lake and Unity Catalog. ·Hands-on Power BI expertise, including data modelling, DAX, RLS and performance optimisation. ·Strong SQL skills and experience with Azure data services. ...

highly visible to senior leadership. You Will Work With Designing and scaling Azure Databricks-based data platforms Building robust data pipelines using Databricks, PySpark, Delta Lake and Airbyte Developing AI and LLM-powered capabilities within customer-facing products Delivering data products that support commercial decision-making Optimising Azure … Advanced Python, SQL and PySpark skills Experience designing and building modern data platforms and pipelines Databricks optimisation, performance tuning and cost management Experience with Delta Lake and Lakehouse architectures Strong Azure cloud experience Experience with modern data ingestion tools such as Airbyte Exposure to Generative AI, LLMs ...

PySpark and SQL Develop reusable ingestion, transformation, validation and enrichment processes Ingest demographic datasets, postcode data, geographic boundaries and reference datasets Build and maintain Delta Lake datasets and manage data lineage Perform postcode enrichment and geographic assignment processing Implement spatial joins, point-in-polygon matching and location-based … criteria Essential Skills & Experience Strong commercial experience with: Databricks Python PySpark SQL Proven experience building and supporting production-grade data pipelines Strong knowledge of Delta Lake, schema management, merge logic and incremental processing Experience handling large datasets within cloud-based environments Practical experience working with geospatial data ...

deployment Worked across monitoring, logging and operational alerting Completed data-quality and reconciliation controls API/file/SFTP integration patterns Nice to have: Delta Lake/Delta MERGE Databricks Asset Bundles Azure DevOps MLOps/model deployment work history SFMC or Comarch integration Customer decisioning/ ...

customer in Central London. Key ResponsibilitiesData Lakehouse Architecture: Design and optimize a high-performance lakehouse pattern utilizing Microsoft Fabric OneLake and Azure Data Lake Storage Gen2 (ADLS Gen2).Data Zone Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely … ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls ...

native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch and streaming patterns Transformation and quality: dbt, Great ...

native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch and streaming patterns Transformation and quality: dbt, Great ...

enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need ...

enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need ...

deadlines. Databases & Tools knowledge: Microsoft SQL Server (administration, T-SQL, DML, DDL, SSIS), Databricks, Power BI.Experience in Azure Technologies: ADLS, ADF, Azure SQL Database, Delta Lake, Azure DevOps, Logic Apps, Function Apps, Key Vaults, Storage Accounts. Infrastructure & Automation: Terraform, Python/PySpark. Expertise in data modeling, integration ...

initiatives.In this role you will· Design and build scalable data pipelines using Databricks, PySpark, SQL, AWS Glue and Lambda.· Develop and manage Data Lake solutions using Bronze, Silver, and Gold data architectures.· Implement batch and real-time data processing using Spark Structured Streaming and Auto Loader.· Optimize Databricks workloads … experience with Databricks, PySpark, SQL, and Apache Spark· Strong experience designing and delivering cloud-based data platforms and lakehouse architectures· Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, and performance tuning· Experience building batch and streaming data pipelines· Strong knowledge of data modelling and data product ...