1 to 25 of 41 Delta Lake Jobs in the UK

pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Solid AWS experience including services such as S3, Glue, Lambda, Batch, and IAM Proficiency ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

with a genuine passion for software engineering in a data context. You'll work across the modern Azure data stack - ADF, ADLS Gen2, PySpark, Delta Lake - with increasing exposure to Microsoft Fabric as the platform matures. You'll collaborate closely with customers and internal teams to ensure data … streaming platforms (e.g. Event Hubs, Kafka), and legacy databases into ADLS Gen2 and OneLake Develop structured data models aligned to Lakehouse, Medallion Architecture, and Delta Lake patterns Continuously profile, debug, and optimise Spark jobs, SQL queries, and Python scripts for maximum performance and cost-efficiency at scale Champion ...

work at home 100% of the time. Your Role: Design, develop, and optimise large-scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog. Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance … continuous improvement. Your Skills: Expert-level hands-on experience with Azure Data Services and Databricks in enterprise, highly regulated environments. Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing ...

elevating engineering capability across engagements. What you'll do Lead the engineering, optimisation and governance of Azure-based data platforms, spanning ADF, Data Lake, Azure Functions, Key Vault, Databricks, Delta Lake, PySpark and Unity Catalog . Collaborate closely with insurance domain teams- Underwriting, Actuarial, Delegated Authority, Bordereaux … efficient operation. What you'll bring Expert-level hands-on experience with Azure data services and Databricks in enterprise-scale environments. Strong knowledge of Delta Lake , Medallion architecture, distributed compute and Lakehouse engineering patterns. Advanced Python, PySpark and Spark SQL skills, including deploying data workloads through Azure DevOps ...

hands-on role. What You’ll Be Doing: • Designing and building end-to-end data pipelines using Databricks (PySpark/SQL/Delta Lake) • Developing and managing robust, scalable Lakehouse architectures • Implementing Delta Live Tables (DLT) for reliable, testable, production-grade pipelines • Optimising performance using: • Working with … Platform teams to improve developer experience and platform usability Key Skills & Experience: • Strong experience building pipelines with PySpark/Spark SQL • Deep understanding of Delta Lake • Databricks Platform Expertise • Hands-on experience with: Unity Catalog (governance & security), Spark Declarative Pipelines, Databricks Workflows/Jobs • Strong understanding of Lakehouse ...

voice interfaces), and the next wave of capabilities depends on a strong data engineering foundation. A Databricks lakehouse POC is being evaluated (Unity Catalog, Delta Sharing, PrivateLink to RDS), and the team is assessing the right long‐term analytics platform strategy. This role builds the entire data engineering function … PostgreSQL, with several modules already migrated. Pipelines must bridge both. • Databricks/Apache Spark — a lakehouse POC is being evaluated with Unity Catalog and Delta Sharing via AWS PrivateLink. Experience with Databricks or equivalent analytics platforms is important for assessing and scaling the right approach • Apache Kafka — event streaming ...

Responsibilities Design and implement data architectures on the Databricks Lakehouse Platform Build and optimize ETL/ELT pipelines using Apache Spark and Databricks Implement Delta Lake and medallion architecture (Bronze, Silver, Gold) Ensure data governance, security, and performance optimization Integrate Databricks with cloud platforms and enterprise data sources … deliver scalable data solutions Required Skills & Qualifications Databricks Certification (e.g., Databricks Certified Data Engineer Associate/Professional) Strong experience with Databricks, Apache Spark, and Delta Lake Proficiency in Python, SQL, and data pipeline development Experience with cloud platforms (Azure, AWS, or GCP) Solid understanding of modern data architecture ...

practices for platform components including Lakehouse architecture, MLOps, and data governance. Platform Engineering Architect and guide implementation of Databricks workspace setup, clusters, Unity Catalog, Delta Lake, workflows, and model serving. Lead the integration of the platform with key internal systems such as Radar, EDP (Enterprise Data Platform … risk assessments. Required Skills & Experience: 8-12+ years of experience in data/AI engineering or cloud architecture roles. Strong expertise with Databricks (Delta Lake, Unity Catalog, MLflow, Workflows). Deep understanding of AWS cloud architecture and services relevant to data/AI workloads. Hands-on experience ...

Databricks, including target architecture, data models, security, and governance frameworks. Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL. Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration process. … business stakeholders to minimize migration risk, ensure smooth cutover, and maintain production stability. Skills Must have Advanced Databricks Experience. Deep knowledge of Databricks architecture, Delta Lake, job orchestration, cluster management, and performance tuning. Proven Migration Experience (Palantir → Databricks). Demonstrated experience leading or executing platform migrations, including pipeline ...

energy market data using Python, SQL and Spark. Support data flows ensuring generation and storage assets deliver the correct contracted schedules. Design and manage Delta Lake tables and SQL databases, applying strong data modelling and optimisation strategies. Develop monitoring dashboards and alerting to ensure transparency around pipeline health … similar production-focused role Strong Python (Pandas) and SQL skills Experience building and maintaining data pipelines Hands-on experience with Spark, Databricks or Delta Lake/Lakehouse architectures Familiarity with cloud data platforms, particularly within Azure environments Good understanding of data modelling, storage formats and performance optimisation Experience ...

DSFS data model and existing Netezza structures. Produce the EDW logical and physical data models, dimensional models for the Semantic Layer, and Data Lake zone architecture (Bronze/Silver/Gold). Define Enterprise Semantic Layer design, including pre-computed aggregations, dimensional models, and Tableau connectivity specifications. Lead source … data architecture, with at least 3 years leading enterprise data model design. Experience designing Lakehouse architecture: medallion architecture (Bronze/Silver/Gold), Delta Lake, or equivalent. Proficiency in dimensional modelling (Kimball methodology) and enterprise data modelling. Experience with Databricks (Unity Catalog, Delta Lake) or Snowflake ...

PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure acceptable) Understanding of data lake and lakehouse architectures Ability to work autonomously and collaboratively within delivery squads Experience working within Central Government departments (highly desirable) Desirable Azure Data Lake, Synapse, Delta Lake Streaming technologies (Kafka, Event Hubs ...

reliable "single version of the truth" for reporting and analytics;- Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS);- Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … Connect).Experience- Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing).- Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach.- Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

mentoring, and support across the data engineering team Cloud Data Platform & Infrastructure • Design and build enterprise-grade data solutions across Azure and GCP (Data Lake, Synapse, BigQuery, event streaming) • Own infrastructure using Infrastructure as Code (Terraform/Bicep) and implement CI/CD pipelines • Ensure platforms are highly available … including: o BigQuery (data warehousing & ELT) o Dataflow (Apache Beam for ETL processing) o Pub/Sub (event streaming ingestion) o Cloud Storage (data lake ingestion layers) o Cloud Composer (Airflow orchestration) • Proven experience building and optimising lakehouse architectures (Delta Lake, Databricks, Snowflake, BigQuery) • Hands-on experience ...

reliability Collaborate with analysts and AI engineers to deliver robust and reusable data products Manage data lakes and warehouses using formats such as Delta Lake and Parquet Implement best practices for data governance, performance, and security Continuously evaluate and adopt new technologies to evolve the organisation's data … platform Provide technical guidance to junior engineers and contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable ...

reliability Collaborate with analysts and AI engineers to deliver robust and reusable data products Manage data lakes and warehouses using formats such as Delta Lake and Parquet Implement best practices for data governance, performance, and security Continuously evaluate and adopt new technologies to evolve the organisation’s data … platform Provide technical guidance to junior engineers and contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable ...

Kimball, Data Vault). Experience exposing Lakehouse architectures to tools such as Power BI and Tableau. Hands-on experience in Databricks engineering: Spark, Delta Lake, PySpark. Skilled in building scalable ETL pipelines using Workflows and Delta Live Tables. Proficiency in Python, Scala, or SQL. Knowledge of Databricks ...

Kimball, Data Vault). Experience exposing Lakehouse architectures to tools such as Power BI and Tableau. Hands-on experience in Databricks engineering: Spark, Delta Lake, PySpark. Skilled in building scalable ETL pipelines using Workflows and Delta Live Tables. Proficiency in Python, Scala, or SQL. Knowledge of Databricks ...

Kimball, Data Vault). Experience exposing Lakehouse architectures to tools such as Power BI and Tableau. Hands-on experience in Databricks engineering: Spark, Delta Lake, PySpark. Skilled in building scalable ETL pipelines using Workflows and Delta Live Tables. Proficiency in Python, Scala, or SQL. Knowledge of Databricks ...

Proven experience delivering enterprise-scale Databricks solutions end-to-end.* Strong background in Lakehouse Architecture, including structured and unstructured data.* Expertise in Spark, PySpark, Delta Lake, and Databricks workflows.* Experience building scalable ETL/ELT pipelines, including Delta Live Tables.* Strong programming skills in Python, Scala ...

Proven experience delivering enterprise-scale Databricks solutions end-to-end.* Strong background in Lakehouse Architecture, including structured and unstructured data.* Expertise in Spark, PySpark, Delta Lake, and Databricks workflows.* Experience building scalable ETL/ELT pipelines, including Delta Live Tables.* Strong programming skills in Python, Scala ...