Data Engineer (City Of London)

Data Engineer

Azure | Databricks | PySpark | Hybrid Cloud | Fabric

Location: London (Hybrid)

Salary: £85,000 + Bonus + Benefits

Type: Permanent

A Data Engineer is required for a fast-evolving (re)insurance business at the heart of the Lloyd's market, currently undergoing a major data transformation. With a strong foundation in the industry and a clear vision for the future, they're investing heavily in their data capabilities to build a best-in-class platform that supports smarter, faster decision-making across the business.

As part of this journey, they're looking for a Data Engineer to join their growing team. This is a hands-on role focused on building scalable data pipelines and enhancing a modern Lakehouse architecture using Databricks , PySpark , and Azure . The environment is currently hybrid cloud and on-prem , with a strategic move towards Microsoft Fabric -so experience across both is highly valued.

What you'll be doing:

• Building and maintaining robust data pipelines using Databricks , PySpark , and Azure Data Factory .
• Enhance and maintain a Lakehouse architecture using Medallion principles
• Working across both cloud and on-prem environments , supporting the transition to Microsoft Fabric .
• Collaborating with stakeholders across Underwriting, Actuarial, and Finance to deliver high-impact data solutions.
• Support DevOps practices and CI/CD workflows in Azure

Technical Requirements:

• Strong hands-on experience with:
• Databricks (including pipeline development and orchestration)
• PySpark and Python for data transformation and processing
• Azure Data Factory for data integration
• Solid understanding of:
• Medallion architecture and Lakehouse design principles
• Hybrid cloud/on-prem environments
• Microsoft Azure ecosystem
• Exposure to or interest in:
• Microsoft Fabric and its evolving role in enterprise data platforms
• Azure DevOps for CI/CD and deployment
• T-SQL and dimensional modelling (Kimball methodology)
• Experience in Financial Services or Lloyd's market is a plus

Apply now or get in touch to find out more - alexh@pioneer-search.com