Lead Azure Databricks Engineer - Insurance (Lloyd's Market)

Base Salary: £78,000-£95,000 Location: Hybrid - London Sector: Data Engineering / Cloud / Insurance Type: Permanent

We are searching for an accomplished Lead Azure Databricks Engineer to join our consulting practice and take technical ownership of complex cloud data engineering projects across the Lloyd's of London insurance market. This position is ideal for someone who is deeply hands-on, highly delivery-driven, and experienced in navigating the regulatory, operational and data challenges unique to the London Market.

Working as part of a specialist consultancy, you'll guide clients through the design, build and optimisation of enterprise-scale Azure data platforms-shaping strategies, solving difficult engineering problems, and elevating engineering capability across engagements.

• Lead the engineering, optimisation and governance of Azure-based data platforms, spanning ADF, Data Lake, Azure Functions, Key Vault, Databricks, Delta Lake, PySpark and Unity Catalog.
• Collaborate closely with insurance domain teams-Underwriting, Actuarial, Delegated Authority, Bordereaux, Exposure Management, Reinsurance, Finance, Risk and Solvency II-delivering solutions aligned to Lloyd's Market standards.
• Translate business, regulatory and operational requirements into scalable and robust cloud data solutions.
• Quickly diagnose and resolve high-complexity technical challenges, providing clear direction and decisive technical leadership.
• Establish and champion best-practice patterns across data lifecycle management, CICD, architecture design principles and cloud-native engineering.
• Provide mentoring, guidance and leadership to engineers across consulting engagements, setting the bar for delivery quality and engineering maturity.
• Identify opportunities for performance improvements, cost reduction, platform resilience and automation.
• Oversee the end-to-end development of data pipelines, ensuring reliability, observability and efficient operation.

• Expert-level hands-on experience with Azure data services and Databricks in enterprise-scale environments.
• Strong knowledge of Delta Lake, Medallion architecture, distributed compute and Lakehouse engineering patterns.
• Advanced Python, PySpark and Spark SQL skills, including deploying data workloads through Azure DevOps CICD, branching/merging and automated testing.
• Solid understanding of data governance, lineage, access controls, FinOps practices and secure cloud engineering.
• Excellent communication skills with the ability to articulate technical topics clearly to senior stakeholders and guide cross-functional discussions.
• A strong delivery focus with a consistent track record of taking ownership and completing major engineering initiatives.
• Experience operating in fast-paced consulting environments with evolving priorities and demanding timelines.
• Essential: hands-on experience working within the Lloyd's of London / London Market insurance sector.

• £78,000-£95,000 base salary
• Hybrid working model based in London
• High-impact consulting role with exposure to top-tier insurance clients
• Opportunity to influence cloud engineering standards and contribute to major data transformation initiatives