Data Engineering Lead

Our client is a large UK housing provider managing over 21,000 homes across England.

The organisation is currently making significant investment in its data and digital capabilities, including the implementation of a modern Microsoft Fabric based data platform and new CRM technologies to improve services across the business.

This role sits at the centre of that transformation, offering the opportunity to help build and lead a brand new data engineering capability within a growing data team.

The Role

We are seeking a Lead Data Engineer to play a key role in developing a modern data platform while leading and mentoring a small engineering team.

Reporting to the Head of Data, you will combine hands on technical delivery with team leadership, helping to design scalable data pipelines, support internal stakeholders, and enable faster insight across the organisation.

This is a unique opportunity to shape how data engineering operates within a business that is early in its data maturity journey.

Key Responsibilities

Leadership & Team Development

• Lead and grow a new data engineering function

• Line manage and mentor two data engineers

• Promote modern engineering practices and tooling

• Establish data engineering as a core organisational capability

Data Platform Development

• Design, build and maintain scalable data pipelines

• Contribute to the development of a Microsoft Fabric data platform

• Work with Lakehouse/medallion architecture

• Develop ETL/ELT pipelines using PySpark and SQL

• Ingest and transform data from SQL systems, flat files, and CSV sources

Collaboration & Stakeholder Engagement

• Partner with teams across the organisation to deliver high-value data products

• Translate technical concepts into clear business language

• Support the organisation's wider data maturity journey

Technical Delivery

• Deliver data solutions that enable faster time-to-value

• Support analytics and reporting teams with reliable, well-structured datasets

• Contribute to data modelling and architectural decisions

Technology Stack

The current data platform includes:

• Microsoft Fabric

• PySpark

• T-SQL

• Lakehouse/Medallion architecture

• Power BI

Current workloads are primarily batch-based, with ambitions to move towards streaming capabilities in the future.

Essential Skills & Experience

• Strong background in data engineering

• Experience building complex data pipelines

• Strong hands-on experience with PySpark and SQL

• Experience with Microsoft Fabric, Azure, or modern cloud data platforms

• Understanding of data modelling and architectural principles

• Experience mentoring or leading engineers

• Strong stakeholder engagement and communication skills

Desirable Experience

• Microsoft certifications (eg DP-900 or Azure Data certifications)

• Experience contributing to data architecture design

• Exposure to machine learning pipelines

• Experience working within Agile delivery environments

The Ideal Candidate

• A hands on technical leader

• Comfortable working in organisations early in their data maturity journey

• Strong mentoring and coaching capabilities

• Able to translate technical delivery into clear business value

• Motivated by the opportunity to build a function from the ground up