1 to 25 of 48 Delta Lake Jobs in the UK

Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus on common IDs, Golden Records … survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity Catalog, data lineage ...

Experience: 610+ Years Key Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus … survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity Catalog, data lineage ...

build of large-scale data platforms using Azure Databricks and modern cloud technologiesImplement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principlesEnhance performance, manage cloud costs and ensure platform reliability for structured streaming workloadsDefine data governance, security and quality standards to maintain … including advanced optimization, data skew mitigation and query tuningStrong programming skills in Python with knowledge of modern software design principlesPractical experience with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake ...

Server and Databricks. Support Lakeflow pipelines, production jobs and data quality processes. Monitor, troubleshoot and optimise data workflows and platform performance. Work with Delta Lake, Unity Catalog and modern data lake architectures. Integrate Azure services including ADLS, Data Factory, Key Vault and REST APIs. Support CI/… enterprise environment. Excellent SQL Server/T-SQL and data engineering skills. Experience building and optimising large-scale data pipelines. Knowledge of Spark, Delta Lake, Unity Catalog and Azure data services. Exposure to Git, CI/CD and infrastructure automation tools. Strong communication skills and a proactive approach ...

data solutions, from conceptual and logical models through to physical implementation Building and maintaining scalable pipelines and lakehouse solutions using Azure Databricks, Spark and Delta Lake Designing and optimising Snowflake environments for performance, scalability and cost Supporting batch, streaming and real-time data processing, including high-volume transactional … environments Hands-on, current engineering skills: you still write the code, not just the diagrams Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake and Lakehouse architecture Strong Snowflake architecture and optimisation experience Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics and/or Microsoft ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

best practices across the enterprise data ecosystem. The role leads the design and delivery of scalable, secure, high-performing data products using Databricks, Delta Lake, Unity Catalog and Azure Data Services The role provides technical leadership, ensuring solutions are robust, cost-effective, highly available, and aligned to business … development. To ensure data dictionaries and models are robust and maintained on a regular basis. Technical Skill Expert-level Databricks experience Advanced PySpark development Delta Lake architecture and optimisation Databricks SQL Data warehouse Dimensional modelling Data Lake Databricks Genie Python development Key Skills, Experience and Knowledge Extensive ...

cost and usage data platform on Databricks, ingesting Azure Cost Management exports, reservation and Savings Plan data, resource inventory and tagging metadata into Delta Lake and modelling it into trusted datasets. Build and own Power BI semantic models behind cost reporting, including DAX measures, composite and DirectQuery designs … rather than simply creating reports. Nice to HaveFinOps or cloud-cost experience, including Azure Cost Management exports, FOCUS, reservations or Savings Plans. Databricks and Delta Lake. KQL.TMDL and XMLA.PowerShell. Experience with cloud cost optimisation, allocation/showback or cost attribution. What We're UsingPython — production services, data pipelines, automation ...

supporting colleagues. What you'll do ·Build and optimise pipelines using Azure Databricks, PySpark, SQL, ADF and ADLS Gen2. ·Develop governed lakehouse solutions using Delta Lake, medallion architecture and Unity Catalog. ·Own Power BI semantic models, DAX measures and row-level security, ensuring reporting is accurate, secure …/CD. ·Mentor colleagues and share good practice across the team. What you'll bring ·Strong production experience with Databricks, PySpark, Delta Lake and Unity Catalog. ·Hands-on Power BI expertise, including data modelling, DAX, RLS and performance optimisation. ·Strong SQL skills and experience with Azure data services. ...

highly visible to senior leadership. You Will Work With Designing and scaling Azure Databricks-based data platforms Building robust data pipelines using Databricks, PySpark, Delta Lake and Airbyte Developing AI and LLM-powered capabilities within customer-facing products Delivering data products that support commercial decision-making Optimising Azure … Advanced Python, SQL and PySpark skills Experience designing and building modern data platforms and pipelines Databricks optimisation, performance tuning and cost management Experience with Delta Lake and Lakehouse architectures Strong Azure cloud experience Experience with modern data ingestion tools such as Airbyte Exposure to Generative AI, LLMs ...

deployment Worked across monitoring, logging and operational alerting Completed data-quality and reconciliation controls API/file/SFTP integration patterns Nice to have: Delta Lake/Delta MERGE Databricks Asset Bundles Azure DevOps MLOps/model deployment work history SFMC or Comarch integration Customer decisioning/ ...

customer in Central London. Key ResponsibilitiesData Lakehouse Architecture: Design and optimize a high-performance lakehouse pattern utilizing Microsoft Fabric OneLake and Azure Data Lake Storage Gen2 (ADLS Gen2).Data Zone Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely … ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls ...

lead engineer experience on enterprise scale Databricks platforms on Azure Strong hands on data engineering background with Python or PySpark and advanced SQL Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows and Auto Loader Declarative pipelines (Delta Live Tables or Lakeflow) and Structured Streaming CI/… Databricks using Databricks Asset Bundles, Azure DevOps or GitHub Actions, and Terraform or Bicep Azure Data Factory, Azure SQL, Data Lake Storage Gen2 Entra ID, Key Vault, private endpoints, RBAC Experience of data governance and lineage, ideally with Microsoft Purview UK GDPR and DPA 2018 in practice, including public ...

level discussion and technical detail Hands on implementation experience, ideally from a data engineering or developer background Databricks at architecture level on Azure, including Delta Lake, Unity Catalog and Spark optimisation Lakehouse and medallion patterns, batch and streaming ingestion, event driven design Conceptual, logical and physical data modelling. … Dimensional and data vault both useful Advanced SQL, Python or PySpark Declarative pipelines (Delta Live Tables or equivalent), Databricks Workflows, Auto Loader and Structured Streaming Microsoft Purview for data governance, cataloguing and lineage Azure Data Factory, Azure SQL, Data Lake Storage Gen2 Azure DevOps or GitHub Actions, Databricks ...

level discussion and technical detail Hands on implementation experience, ideally from a data engineering or developer background Databricks at architecture level on Azure, including Delta Lake, Unity Catalog and Spark optimisation Lakehouse and medallion patterns, batch and streaming ingestion, event driven design Conceptual, logical and physical data modelling. … Dimensional and data vault both useful Advanced SQL, Python or PySpark Declarative pipelines (Delta Live Tables or equivalent), Databricks Workflows, Auto Loader and Structured Streaming Microsoft Purview for data governance, cataloguing and lineage Azure Data Factory, Azure SQL, Data Lake Storage Gen2 Azure DevOps or GitHub Actions, Databricks ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise data architecture, balancing … growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ELT solutions. ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise data architecture, balancing … growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ELT solutions. ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise data architecture, balancing … growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ELT solutions. ...

engineering within cloud-based environments. Advanced Python and SQL development skills. Hands-on experience with Azure Databricks, Apache Spark and PySpark. Good understanding of Delta Lake, lakehouse architecture and modern data modelling. Experience building and managing production data pipelines, including batch and streaming processes. Knowledge of Azure services ...

Office + 2 Days RemotePay: c.700 per day PAYEConsultant: Joe MooreThis is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company. Role Description: We are looking for a highly skilled Senior Data Engineer with strong analytical capabilities … scalable data pipelines and enabling real-time and batch analytics across the organization. Key Responsibilities: Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)Develop and manage streaming data solutions for real-time analytics and decision-makingIngest data from various sourcesincluding SAP systems using Qlik ...

Databricks Define and enforce platform standards, ensuring compliance with data governance and security policies Build and optimize Databricks streaming workloads, including Structured Streaming and Delta Live Tables Collaborate with cross-functional teams to translate requirements into robust technical solutions Drive platform performance tuning, cost optimization and monitoring strategies … 3+ years in a tech lead capacity Strong hands-on expertise in Databricks for enterprise-scale workloads Proficiency in PySpark, Spark Structured Streaming, and Delta Lake Solid programming skills in Python and demonstrated SQL performance optimization experience Deep understanding of Azure Data Platform services and cloud-native engineering ...

scalable, secure and high-performing solutions that support analytics, AI, and enterprise reporting. Key Skills Required AWS (S3, Glue, Redshift, Athena, Kinesis, Lambda) Data Lake & Lakehouse architectures ETL/ELT & data pipelines Python, SQL & PySpark Data modelling (Kimball, Data Vault) Data governance & security Terraform/CloudFormation Real-time & batch … best practices Working closely with business and technical stakeholders Leading data transformation and cloud migration initiatives Desirable Databricks, Snowflake, dbt Kafka, Apache Iceberg, Delta Lake Data Mesh/Data Fabric AI & ML-enabled data platforms ...

Design, build and enhance a secure, scalable and reliable Databricks platformDevelop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta LakeContribute to platform architecture, engineering standards and best practices across the data estateWork with Product, Analytics and Data Science teams to deliver effective data … continuous improvement across the engineering teamJob Requirements: Significant experience building and supporting scalable, production-grade data platformsStrong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQLGood understanding of cloud-based data engineering, ideally within Microsoft Azure environmentsExperience with Infrastructure as Code tools such as Terraform ...

interpersonal skills - you communicate clearly with stakeholders as well as other engineers, fostering a collaborative, supportive working environmentExperience in the financial markets, especially in delta one, store of value, and/or FICC options tradingExperience with Linux-based, concurrent, high-throughput, low-latency software systemsExperience with pipeline orchestration frameworks … e.g. Airflow, Dagster)Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databasesHave a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experienceFor more information about DRW's processing activities ...

secure handling of inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena ...