6 of 6 Delta Lake Jobs in the City of London

platforms on Azure. 💡 Why this role stands out Architect the core data platform for a rapidly evolving, tech-driven insurance environment Work with Databricks, Delta Lake, and Azure SQL at scale Drive innovation across data, AI, and platform engineering 🔧 What you’ll be doing Own and evolve enterprise … data architecture (warehouse + lakehouse) Design scalable data models (dimensional, canonical, star/snowflake) Build and optimise Databricks solutions (Delta Lake, DLT, SQL Warehouses) Set standards for performance, governance, and cost efficiency on Azure SQL Lead best practices across ELT/ETL pipelines, CI/CD, and reusable ...

real-time insights. Key Responsibilities Architecture & Solution Design Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures … leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata ...

Databricks, including target architecture, data models, security, and governance frameworks. Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL. Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration process. … business stakeholders to minimize migration risk, ensure smooth cutover, and maintain production stability. Skills Must have Advanced Databricks Experience. Deep knowledge of Databricks architecture, Delta Lake, job orchestration, cluster management, and performance tuning. Proven Migration Experience (Palantir → Databricks). Demonstrated experience leading or executing platform migrations, including pipeline ...

reliability Collaborate with analysts and AI engineers to deliver robust and reusable data products Manage data lakes and warehouses using formats such as Delta Lake and Parquet Implement best practices for data governance, performance, and security Continuously evaluate and adopt new technologies to evolve the organisation’s data … platform Provide technical guidance to junior engineers and contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable ...

equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics * Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage; infrastructure as code for data workloads (Terraform or Bicep) * Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs ...

documentation Collaborate with stakeholders to deliver analytics-ready datasets Required Skills Strong experience with Databricks , SQL, and Python (Pandas) Solid understanding of Spark, Delta Lake, and Databricks tools (DLT, Workflows, Unity Catalog) Expertise in data modelling : Star Schema, SCD, CDC Experience with dbt, Azure DevOps, CI/ ...