Data Platform Lead Engineer (SEL & SWL)

Job description

Job responsibilities

Who are we looking for?

We are looking for an accomplished Data Platform Engineer with substantial technical depth and experience designing, building and operating complex data platforms, pipelines and analytical data models at scale.You will understand the complete data lifecyclefrom ingestion and orchestration through transformation, modelling, validation, deployment, monitoring and optimisationand be able to determine the appropriate engineering approach for complex analytical requirements.

You will combine engineering expertise with strategic and analytical judgement. You will be comfortable interrogating architecture and code, challenging technical approaches, identifying technical debt and establishing engineering standards that improve reliability, maintainability, scalability and reuse.Your experience is likely to include several of the following:

Microsoft Fabric Snowflake Azure OneLake Lakehouse architecture Data Factory Spark Python SQL ETL/ELT dimensional modelling Data Vault orchestration Git Azure DevOps/GitHub CI/CD Infrastructure as Code data quality MLOps performance optimisationYou will also be pragmatic and highly collaborative. You will understand the problem before designing the solution, communicate complex engineering concepts clearly and take ownership of difficult technical problems from initial design through to stable production operation.Above all, we are interested not simply in what you have built, but what became measurably better because you built it.

What makes the role different?

For us, successful Data Engineering is not simply about moving data from one place to another. The defining proposition of this role is to engineer for analytical value. We want to move away from unnecessary manual processing and isolated technical solutions towards automated, reusable, governed, integrated and production-quality data products.You will work directly with the analysts and Data Scientists who consume the data, enabling you to understand how engineering decisions affect analytical delivery and target technical effort where it can make the greatest difference.

You will have the authority and technical credibility to challenge existing approaches, improve patterns for ingestion, orchestration, transformation and modelling, and ensure engineering solutions are appropriately designed, tested, monitored and optimised.

Success will therefore be visible and measurable. It could mean reducing a data preparation process from hours to minutes; eliminating repetitive analyst intervention; improving data quality or pipeline reliability; increasing reuse of curated datasets and common transformation logic; reducing compute consumption; shortening the time required to develop analytical products; or releasing analytical capacity for higher-value work.

Ultimately, we want someone who can connect engineering excellence with measurable improvements in analytical delivery.

What will you do?

1. Lead the design, build and optimisation of scalable data pipelines and architecture create resilient, secure and high-performing data flows from source systems through ingestion and transformation to curated analytical data products.
2. Provide technical leadership and engineering assurance for data orchestration and transformation set and challenge engineering standards and ensure ETL/ELT solutions are automated, maintainable, observable, appropriately tested and engineered for performance, resilience and reuse.
3. Lead and technically assure analytical data modelling work directly with analysts and Data Scientists to translate complex analytical requirements into robust, reusable data structures and semantic models, constructively challenging modelling approaches where better solutions are available.
4. Drive automation across Data Engineering and the wider analytical function identify repetitive, manual and inefficient processes and develop engineering solutions that release capacity, increase reuse and improve analytical productivity.
5. Establish strong software and Data Engineering disciplines embed automated testing, Git-based version control, CI/CD, deployment pipelines, monitoring, documentation, change control and Infrastructure as Code.
6. Engineer for reliability, resilience and data quality establish validation, reconciliation, exception handling, recovery, backfill, monitoring and audit processes so analytical teams can confidently rely on the data they receive.
7. Optimise Microsoft Fabric, Snowflake and Azure workloads apply strong technical judgement to performance, scalability, resilience, capacity consumption and cloud cost optimisation.
8. Enable advanced Analytics, Data Science and AI create the pipelines, feature engineering, data structures and MLOps capabilities required to move analytical and Machine Learning solutions reliably into production.

These responsibilities retain the central requirements of the Band 8B JD: senior responsibility for pipelines and infrastructure, technical architecture, transformation, analytical modelling, testing, automation, optimisation, Data Science enablement and collaboration with analytical teams.

Why join us?

You will work within an integrated Intelligence & Insights function supporting South East and South West London, alongside Data Engineers, Data Scientists, Population Health specialists and wider analytical teams working with complex health and care data.Your engineering will directly support Population Health Management, strategic commissioning, planning, performance and wider analytical delivery, providing the opportunity to see how technical decisions translate into practical analytical capability.You will be joining at an important point in the development of our data environment, with the opportunity to work across Microsoft Fabric, Snowflake and Azure while helping establish the technical foundations for faster, more reliable and scalable analytical delivery.If you are a technically accomplished Data Platform Engineer who combines engineering depth, strategic thinking, analytical curiosity and collaboration with a determination to turn engineering excellence into measurable analytical value, we would like to hear from you.

1. Lead the design, build and optimisation of scalable data pipelines and data architecture ensuring reliable, secure and efficient data flows from source systems through ingestion and transformation to analytical data products.
2. Lead data orchestration, transformation and integration develop robust, automated ETL/ELT processes covering ingestion, cleansing, transformation, scheduling, dependency management, exception handling and delivery.
3. Design and assure robust analytical data models translate complex analytical and business requirements into well-engineered, reusable dimensional, Lakehouse, Data Vault and semantic models that improve analytical consistency and performance.
4. Drive automation and analytical productivity identify manual, repetitive and inefficient processes within Data Engineering, Analytics and the wider department and develop automated solutions that demonstrably release capacity and accelerate analytical delivery.
5. Provide senior technical leadership and engineering assurance set and champion standards for architecture, code, pipelines, data models, testing, documentation and engineering practices, constructively challenging solutions where greater rigour or efficiency is required.
6. Establish modern engineering and deployment practices embed Git, CI/CD, automated testing, monitoring, Infrastructure as Code, deployment pipelines and effective change control to create reliable and maintainable production services. The person specification specifically requires significant DevOps, Git and automated deployment experience.
7. Demonstrate measurable benefit from Data Engineering establish and evidence improvements such as reduced data-processing time, fewer manual interventions, improved pipeline reliability, greater reuse of data assets and transformation logic, faster analytical turnaround and increased analytical capacity.
8. Engineer data quality, reliability and resilience into the platform implement validation, reconciliation, monitoring, audit, recovery, backfill and exception-management processes so analytical teams can confidently rely on the data they receive.
9. Optimise Microsoft Fabric, Snowflake and Azure environments improve performance, scalability, resilience and resource utilisation across OneLake, Lakehouse, Data Factory, Spark/Notebooks, Snowflake and associated cloud services.

Person Specification

Qualifications

• Educated to master's level or equivalent level of experience working at a senior level in a specialist area, with additional in-depth professional knowledge and continuous professional development in own discipline
• Evidence of continued professional development

• Relevant certifications in Cloud & Platform systems, data architecture and design and related subjects

Experience needed - general or specific

• Extensive experience in Azure infrastructure design, deployment and management, ideally in an NHS setting
• Extensive experience designing, building and maintaining complex ETL/ELT pipelines at scale, including automated scheduling and or-chestration
• Significant knowledge and experience of lead-ing the integration of dozens of source sys-tems
• Significant knowledge and experience for re-covery and backfill processes
• Demonstrable experience deploying, configuring and managing MS Fabric environments, including Lakehouse, Data Factory, Spark/Notebooks and Power BI integration
• Significant knowledge and experience of performance optimisation
• Significant experience in dimensional model-ling, data vault and lakehouse patterns, in-cluding slowly changing dimensions and event-driven pipelines
• Proven experience leading data quality frame-works including validation, reconciliation, monitoring and incident management
• Knowledge and experience of GDPR, encryption
• Experience implementing Identity & Access Management, Role-based Access Controls, data masking, row-level security and audit loggingKnowledge and experience of Masking and Tokenisation
• Significant experience in infrastructure as code, DevOps practices, Git integration, branching strategies and automated deploy-ment pipelines
• Knowledge and experience on developing machine learning pipelines
• Knowledge and experience of Machine learn-ing Ops
• Demonstrable experience in maintaining an enterprise level Snowflake environment, in-cluding credit management, RBAC, and secu-rity governance, ideally in an NHS setting

• Knowledge and experience of agile product operating models of delivery
• Knowledge and experience of Machine learning Ops

SNOWFLAKE

• Significant knowledge and experience of the three-layer architecture
• Significant knowledge and experience of Vir-tual warehouses, including multi-cluster ware-houses, workload isolation
• Significant knowledge and experience of data ingestion and integration: bulk loading, contin-uous ingestion, connectors
• Significant knowledge and experience in data transformation & modelling: ELT patterns, dbt + Snowflake integration, stored procedures, star schema, building curate layer, incremen-tal models, slowly changing dimensions
• Significant experience and knowledge of per-formance & cost optimization: clustering, mi-cro-partitions, credit consumption, resource monitoring, warehouse sizing, query profile, account usage views, schema information
• Significant knowledge and experience of se-curity, governance & compliance: access con-trols, data protection, compliance
• Significant knowledge and experience of ad-vance enterprise capabilities: data sharing, Snowpark, native apps, unistore and iceberg tables
• Significant experience of automated deploy-ments, zero-downtime change, disaster re-covery

What the service users expect of this post holder

• Understanding performance and change management, business planning, budget setting and monitoring, and contracting processes, in a health setting.
• Demonstrable understanding of managing the delivery and improvement of services within a complex and diverse organisation.
• Demonstrable understanding of leading teams, setting strategic and operational objectives, designing personal development plans, and resolving conflict situations
• Demonstrable knowledge of development of strategies to meet changing organisa-tional needs and acting as the technical subject matter expert across an entire di-rectorate or Organisation.
• Demonstrable understanding of designing, implementing and managing complex modern information and analytics sys-tems, including Cloud-based data ware-houses
• Demonstrable understanding of deploying, configuring, fine tuning, and managing MS Fabric environments and components
• Demonstrable understanding of liaising with ICT, IG, cyber security and other re-lated disciplines and internal and external teams to ensure their timely cooperation on projects that require their input or sign off
• Demonstrable understanding of reviewing and amending large volumes of complex code and identifying performance issues and/or bugs checking for adherence to coding best practice and local standards,
• Demonstrable understanding of working under pressure and in uncertain and highly politically sensitive environments
• Strong leadership influencing, negotiating and interpersonal skills
• A collaborative management style, which rec-ognizes and values personal contributions and agenda and encourages team working
• Knowledge of current and evolving healthcare policies and digital strategies in London and nationally, and good understanding of their im-plications for South London
• Robust knowledge of NHS IG principles and practical understanding of how to achieve compliance with legal requirements, particularly in relation to the processing of sensitive information

• Demonstrable understanding of inter-agency working and the development of joint projects, working with stakeholders to define success criteria, improving performance, and maximize efficiencies
• Demonstrable understanding of leading the planning of complex technical projects, such as the adoption of innovative solutions and the migration across different analytics systems
• Demonstrable understanding of leading the planning of complex technical projects, such as the adoption of innovative solu-tions and the migration across different analytics systems
• Knowledge of current and evolving healthcare policies and digital strategies in London and nationally, and good understanding of their im-plications for South London

Skill and Ability

• The ability to perform system & architectural thinking as it relates to platform architecture, data storage, workload separation, stability design and resilience.
• The ability to undertake query tuning, warehousing sizing, resource monitoring and capacity planning
• The ability to identify, define, promote, com-municate and achieve clear organisational val-ues and goals, effective management pro-cesses, and rational and timely decision taking
• The ability to work in partnership with lead commissioners, clinicians and health care professionals within and outside the ICB and a partnership approach to working across organisations.
• The ability to analyse complex issues, to think and plan to achieve both tactical and strategic objectives, and to exercise sound judgement in the face of conflicting pres-sures.
• The skills to evaluate and learn from out-comes, with a clear commitment to innovation, learning and improvement.
• Effectiveness in conflict resolution and man-agement.
• Ability to provide and receive highly complex, sensitive, confidential, or contentious infor-mation to a range of technical and non-tech-nical stakeholders, including senior manage-ment, negotiating and influencing them on diffi-cult subjects.
• Problem solving skills and ability to prioritise and respond effectively to unplanned and con-flicting demands.
• Autonomous strategic thinking abilities, to an-ticipate and mitigate against problems, formu-lating and adjusting plans as required, and briefing senior management whenever appro-priate.

• The ability to enable analytics and AI through the following: feature engineering pipelines, machine learning prep, MLOps, real-time data design

MS Fabric

• Significant knowledge and experience of OneLake, Fabric workspaces and capacities
• Significant knowledge and experience of data factory: complex orchestration, parameterization, event-based triggers, retry and failure handling, meta-driven ingestion
• Significant knowledge and experience of Lakehouse: delta lake, table optimisation, travel time, schema evolution
• Significant knowledge and experience of Spark/Notebooks and warehousing
• Significant knowledge and experience on how Fabric integrates with power BI semantic models, Directlake mode, DirectQuery
• Significant knowledge and experience of Identity & Access, data security, compliance
• Significant knowledge and experience of performance and cost engineering: query optimisation, file sizing, delta compaction, caching, capacity load patterns, capacity pooling.
• Significant knowledge and experience of Kimball vs data Vault vs Lakehouse, dimensional modelling, slowly changing dimensions and event-driven pipelines

• Significant knowledge and experience DevOps: Fabric and Git integration, AzureDevOps/GitHub, branching strategy deployment pipelines and infrastructure as code
• Knowledge experience in administration and configuration of GitHub and Azure DevOps code repositories for multi team use

Disclosure and Barring Service Check

This post is subject to the Rehabilitation of Offenders Act (Exceptions Order) 1975 and as such it will be necessary for a submission for Disclosure to be made to the Disclosure and Barring Service (formerly known as CRB) to check for any previous criminal convictions.

Employer details

Employer name

South West London Integrated Care Board

Address

120 The Broadway

Wimbledon

SW19 1RH

United Kingdom

Employer's website

https://www.southwestlondon.icb.nhs.uk