1 to 25 of 31 Delta Lake Jobs in the UK

pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Proficiency in Python and SQL for data engineering and analytics TO BE CONSIDERED Please either ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

understanding of cloud security, access control, and governance principles Strong software engineering fundamentals and automation mindset Nice-to-have: Enterprise-scale data platform experience Delta Lake, Delta Live Tables (DLT), and Databricks SQL Azure monitoring and observability tooling Event-driven or streaming architectures Experience working across platform ...

Microsoft Azure and Databricks Key Responsibilities: Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience: Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

reliable 'single version of the truth' for reporting and analytics; - Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS); - Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … . Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing). - Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach. - Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

engineering processes. Mentor junior engineers and support knowledge-sharing across teams. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake. Develop efficient ETL/ELT workflows to process large volumes of structured and unstructured data. Implement data governance, security, and compliance standards. Work … pipeline deployment and monitoring. What We Are Looking For: 5+ years of experience in data engineering or related roles. Strong expertise in Databricks, Spark, Delta Lake, and cloud data platforms (AWS, Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

with OIDC federation) and/or Azure DevOps, supporting trunk-based development practices. Must Have Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation ...

reliability within production data environments. Experience working with cloud data services, ideally Azure, although AWS or GCP experience would also be relevant. Exposure to Delta Lake, Databricks Workflows, scheduling tools or lakehouse environments would be beneficial. An understanding of Power BI, reporting data models, governance, cataloguing or lineage ...

data access .Ensure performance, scalability, and security of backend systems .Databricks & Data Engineerin gDevelop ETL pipelines and workflows using PySpark on Databricks .Work with Delta Lake, Unity Catalog, and Databricks REST APIs .Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS) .SQL Database Managemen ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

connectivity issues across Foundry, Databricks and AWS. Desirable Skills Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories. Experience with Delta Lake, Parquet, Iceberg, CSV and JSON. Data governance, lineage, secure data-sharing and access approval experience. Python, PySpark or SQL development experience. CI/ ...

defining target-state architectures, and delivering scalable, secure, and governed data solutions. Key Responsibilities Design and implement modern lakehouse architectures using Azure Databricks and Delta Lake. Define and establish Medallion Architecture (Bronze, Silver, Gold) data layers to support analytics, reporting, and operational use cases. Configure and manage Unity Catalog … including metastore setup, catalog and schema design, governance policies, and access controls. Architect and manage Azure Data Lake Storage Gen2 environments, including storage containers, security models, lifecycle management, and integration with Databricks. Implement and optimize Microsoft Fabric integrations, including Databricks Unity Catalog mirroring and Power BI connectivity through Databricks ...

connectivity issues across Foundry, Databricks and AWS. Desirable Skills Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories. Experience with Delta Lake, Parquet, Iceberg, CSV and JSON. Data governance, lineage, secure data-sharing and access approval experience. Python, PySpark or SQL development experience. CI/ ...

code methodologies. Required Qualifications 7+ years of experience in Data Architecture or related leadership roles. Strong hands-on experience with Databricks , including Lakehouse Architecture, Delta Lake, Unity Catalog, and Medallion Architecture. Proven expertise in Salesforce Data Architecture , including data modeling, object relationships, integration patterns, and CRM data design. ...

Looking For Proven experience delivering large-scale Databricks transformation projects Strong Financial Services or enterprise consulting background Deep expertise across Azure, Spark, Delta Lake and modern data engineering practices Experience building and leading high-performing technical teams Strong commercial awareness and stakeholder management experience Ability to lead complex ...

Commercial teams to deliver scalable solutions What you bring Must-haves 8+ years in data engineering/data architecture Deep experience with Databricks (Lakehouse, Delta Lake, Unity Catalog) Expert SQL and distributed computing knowledge Strong experience with data modelling (star/snowflake schemas) Proven experience migrating legacy data ...

driven analytics for traders, including seasonality, correlations, regression, forecasting, and scenario modelling. Build scalable, reusable data pipelines in Databricks using PySpark, Spark SQL, Delta Lake, and Unity Catalog. Perform statistical and econometric analysis on large market and fundamentals time‐series datasets. Work directly with traders and analysts … guardrails). Productionise solutions with testing, observability, versioning, documentation, and CI/CD. Must‐Have Skills: Hands‐on Databricks and Spark expertise (PySpark, SQL, Delta, Unity Catalog). Strong data engineering background: ingestion, modelling, orchestration, performance tuning. Solid statistics/econometrics for market time series. Experience building and integrating ...

/CD, and data quality • Building reusable patterns that elevate the whole practice What we're looking for: • Deep, hands-on Databricks expertise (Lakehouse, Delta Lake, DLT, Workflows) • Strong data engineering background — PySpark, Python, SQL, Spark • Cloud experience on Azure, AWS, or GCP • Confident stakeholder management ...

multi-site business environment Strong proficiency in Python and SQL, with hands-on experience in Spark/PySpark and modern table formats (e.g. Delta Lake, Iceberg, Hudi) Expertise in AWS data services (e.g. S3, Glue, Redshift, Lambda, Step Functions, Kinesis) and infrastructure-as-code (Terraform or CDK) Experience ...

reporting. Core requirements: Strong experience delivering end-to-end data engineering solutions in production Deep hands-on expertise with Databricks (jobs, notebooks, workflows, Delta Lake) Experience designing and operating CI/CD pipelines for data platforms Strong Python & SQL skills for scalable data processing Solid understanding of lakehouse ...

data governance frameworks, metadata management, and regulatory compliance It would be a bonus if you have: Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, or data observability tooling Experience leading architecture workshops with both technical and non-technical stakeholders A background providing technical mentorship ...

Modelling, Data Warehousing. Strong background in incident resolution, requests, changes and problem-solving within SLAs. Hands-on experience with Spark, Data Architecture, SQL and Delta Lake. Cloud development experience (AWS, GCP or Azure). Working knowledge of Medallion Architecture. Demonstrated capability in implementing and supporting pipelines in demanding environments. ...

Modelling, Data Warehousing. Strong background in incident resolution, requests, changes and problem-solving within SLAs. Hands-on experience with Spark, Data Architecture, SQL and Delta Lake. Cloud development experience (AWS, GCP or Azure). Working knowledge of Medallion Architecture. Demonstrated capability in implementing and supporting pipelines in demanding environments. ...