23 of 23 Delta Lake Jobs in London

platforms on Azure. 💡 Why this role stands out Architect the core data platform for a rapidly evolving, tech-driven insurance environment Work with Databricks, Delta Lake, and Azure SQL at scale Drive innovation across data, AI, and platform engineering 🔧 What you’ll be doing Own and evolve enterprise … data architecture (warehouse + lakehouse) Design scalable data models (dimensional, canonical, star/snowflake) Build and optimise Databricks solutions (Delta Lake, DLT, SQL Warehouses) Set standards for performance, governance, and cost efficiency on Azure SQL Lead best practices across ELT/ETL pipelines, CI/CD, and reusable ...

reference data in relational databases and Databricks lakehouse tables. 5. Databricks Integration — Leverage Databricks as a core data platform — Unity Catalog for data governance, Delta Lake for reliable storage, Databricks Vector Search for managed embeddings, and Spark/SQL for large-scale data processing and feature engineering. 6. … Understanding of ontology design and entity resolution for financial entities (counterparties, instruments, legal entities). Structured Data & Databricks (must-have) ● Production experience with Databricks — Delta Lake, Unity Catalog, Spark SQL, and ideally Databricks Vector Search or Databricks Feature Serving. ● Experience with text-to-SQL approaches — using LLMs ...

elevating engineering capability across engagements. What you'll do Lead the engineering, optimisation and governance of Azure-based data platforms, spanning ADF, Data Lake, Azure Functions, Key Vault, Databricks, Delta Lake, PySpark and Unity Catalog . Collaborate closely with insurance domain teams- Underwriting, Actuarial, Delegated Authority, Bordereaux … efficient operation. What you'll bring Expert-level hands-on experience with Azure data services and Databricks in enterprise-scale environments. Strong knowledge of Delta Lake , Medallion architecture, distributed compute and Lakehouse engineering patterns. Advanced Python, PySpark and Spark SQL skills, including deploying data workloads through Azure DevOps ...

real-time insights. Key Responsibilities Architecture & Solution Design Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures … leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata ...

readiness, and regulatory compliance. Your responsibilities: Design and govern enterprise data architectures using Microsoft Foundry, Azure Data Factory, ADLS, Synapse, and Databricks (Spark & Delta Lake). Lead data modelling initiatives including star schema and lakehouse patterns. Architect end-to-end ETL/ELT pipelines and integration strategies. Ensure … compliance in regulated environments. Your ProfileEssential skills/knowledge/experience: Strong experience with Microsoft Foundry, Azure (Data Factory, ADLS, Synapse) and Databricks (Spark, Delta Lake). Proficient in Python and SQL. Proven track record designing end-to-end data pipelines and architectures. Deep understanding of ETL/ ...

Databricks, including target architecture, data models, security, and governance frameworks. Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL. Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration process. … business stakeholders to minimize migration risk, ensure smooth cutover, and maintain production stability. Skills Must have Advanced Databricks Experience. Deep knowledge of Databricks architecture, Delta Lake, job orchestration, cluster management, and performance tuning. Proven Migration Experience (Palantir → Databricks). Demonstrated experience leading or executing platform migrations, including pipeline ...

reliability Collaborate with analysts and AI engineers to deliver robust and reusable data products Manage data lakes and warehouses using formats such as Delta Lake and Parquet Implement best practices for data governance, performance, and security Continuously evaluate and adopt new technologies to evolve the organisation's data … platform Provide technical guidance to junior engineers and contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable ...

reliability Collaborate with analysts and AI engineers to deliver robust and reusable data products Manage data lakes and warehouses using formats such as Delta Lake and Parquet Implement best practices for data governance, performance, and security Continuously evaluate and adopt new technologies to evolve the organisation’s data … platform Provide technical guidance to junior engineers and contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable ...

Proven experience delivering enterprise-scale Databricks solutions end-to-end.* Strong background in Lakehouse Architecture, including structured and unstructured data.* Expertise in Spark, PySpark, Delta Lake, and Databricks workflows.* Experience building scalable ETL/ELT pipelines, including Delta Live Tables.* Strong programming skills in Python, Scala ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics * Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage; infrastructure as code for data workloads (Terraform or Bicep) * Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs ...

target mapping, data quality rule setup for the target data model’s critical data elements (CDE). Database Optimization: Optimize Cloud databases such as Delta Lake or One Lake for efficiency, scalability, and faster access using techniques like indexing and partitioning. Data Governance: Work with Federated Governance ...

Bronze/Silver/Gold). Developing scalable models and Power BI semantic layers for enterprise reporting. Enhancing and supporting components across Azure Data Lake, ADF, Synapse, Azure SQL and related Microsoft stack tools. Applying engineering discipline across testing, code management, CI/CD and performance optimisation. Working closely … alignment with architectural standards. What You'll Need to Succeed Strong hands-on engineering experience within Microsoft Fabric (Lakehouse, Pipelines, Dataflows Gen2, Notebooks, Direct Lake). Previous experience with Azure Databricks and Delta Lake engineering. Solid working knowledge of the Microsoft data stack: Azure Data Factory, ADLS ...

documentation Collaborate with stakeholders to deliver analytics-ready datasets Required Skills Strong experience with Databricks , SQL, and Python (Pandas) Solid understanding of Spark, Delta Lake, and Databricks tools (DLT, Workflows, Unity Catalog) Expertise in data modelling : Star Schema, SCD, CDC Experience with dbt, Azure DevOps, CI/ ...

industries A supportive, down-to-earth team What they're looking for: Solid experience with Databricks (Azure or AWS) Good understanding of tools like Delta Lake, Unity Catalog, MLflow Strong SQL and Spark (Python or Scala) Experience working with data pipelines and modern data platforms A good grasp ...

dataPerformance and attribution reportingRisk and analytics outputsRegulatory and trustee disclosuresData governance and operational controlsYou’ll also oversee the migration of structured datasets from Databricks (Delta Lake/Spark) and replace manual Excel workflows with automated, well-governed pipelines to meet audit and regulatory standards.Key ResponsibilitiesBuild and maintain ...

streamlined data analysis, product builds and reduced time to market Competencies and Behaviours 1 -3 years analytics/data science experience Practical knowledge of; Delta Lake architecture and versioned datasets, Data pipelines, orchestration and scheduling concepts Proficiency in analytical programming language such as python and/ ...

using the below technologies: Python, SQL and dbt Lightdash as our BI tool & semantic layer Streamlit for interactive data applications Databricks data platform with Delta Lake Airflow for orchestration Kubernetes for application deployment CI/CD with GitHub and CircleCI Infrastructure on AWS, deployed with Terraform/Spacelift ...

using the below technologies: Python, SQL and dbt Lightdash as our BI tool & semantic layer Streamlit for interactive data applications Databricks data platform with Delta Lake Airflow for orchestration Kubernetes for application deployment CI/CD with GitHub and CircleCI Infrastructure on AWS, deployed with Terraform/Spacelift ...

components to the position: Define the long-term technical vision and roadmap for our modern Data Platform, Mobius (including Data Warehousing, Data Lake, Streaming, and Governance tooling). Design and govern the reference architecture for core data infrastructure, ensuring optimal scalability, reliability, and security. Act as a hands … software and data engineering, with provable experience operating at a Principal, Staff, or equivalent level. Deep proficiency across the modern data stack (Snowflake, BigQuery, Delta Lake, Iceberg, Kafka, Flink). Proven track record designing scalable, self-service data platforms using cloud-native services (AWS/GCP) and infrastructure ...

providing insight, documentation and explanation of new features and changes. Knowledge & Technical Skills Strong understanding of modern data platforms, particularly Databricks Practical knowledge of: Delta Lake architecture and versioned datasets Databricks notebooks (SQL and Python) Data pipelines, orchestration and scheduling concepts Solid SQL skills with the ability ...

based development workflows Experience delivering features end-to-end in distributed teams Confident troubleshooting production issues in real time Desirable Skills Apache Spark (EMR) & Delta Lake PostgreSQL (Aurora) CI/CD using GitHub Actions IaC (Helm, CloudFormation) Monitoring tools (Sentry) Role Responsibilities You'll be hands ...

based development workflows Experience delivering features end-to-end in distributed teams Confident troubleshooting production issues in real time Desirable Skills Apache Spark (EMR) & Delta Lake PostgreSQL (Aurora) CI/CD using GitHub Actions IaC (Helm, CloudFormation) Monitoring tools (Sentry) Role Responsibilities You'll be hands ...