Data Platform Senior Engineer (SEL & SWL) x2

Job description

Job responsibilities

Who are we looking for?

We are looking for an experienced Data Platform Engineer who can design, build, test and maintain modern data pipelines and analytical data models.You will understand the data lifecycle from ingestion and orchestration through transformation, modelling, validation, deployment, monitoring and optimisation. You will be comfortable investigating problems, improving existing solutions and working with senior engineers to apply appropriate engineering standards.Your experience is likely to include several of the following:Microsoft Fabric Snowflake Azure OneLake Lakehouse architecture Data Factory Spark Python SQL ETL/ELT dimensional modelling Data Vault orchestration Git Azure DevOps/GitHub CI/CD Infrastructure as Code data quality MLOps performance optimisationYou will be pragmatic and collaborative, able to understand the problem before developing the solution and communicate technical concepts clearly to both technical and non-technical colleagues.Above all, we are interested not simply in what you have built, but what became better because you built it.

What makes the role different?

For us, successful Data Engineering is not simply about moving data from one place to another. We want our engineers to understand how the data they build and maintain is ultimately used.You will work closely with the analysts and Data Scientists who consume the data, giving you the opportunity to connect engineering decisions directly to analytical delivery.You will help us move away from unnecessary manual processing and isolated solutions towards automated, reusable, governed and production-quality data products. The difference you make should be visible: reducing processing time, eliminating repetitive analyst intervention, improving data quality and pipeline reliability, increasing reuse, improving performance or shortening the time needed to develop analytical products.The role therefore provides an opportunity to develop deep technical expertise while contributing directly to measurable analytical value.

What will you do?

1. Design, build and optimise scalable data pipelines develop secure and reliable data flows from source systems through ingestion and transformation to curated analytical data products.
2. Develop data orchestration, transformation and integration solutions build and maintain automated ETL/ELT processes covering ingestion, cleansing, transformation, scheduling, dependency management and exception handling.
3. Build robust analytical data models translate analytical and business requirements into reusable dimensional, Lakehouse, Data Vault and semantic models.
4. Automate manual and repetitive processes identify opportunities to reduce unnecessary intervention and improve the productivity of Data Engineering and analytical teams.
5. Apply modern engineering disciplines use Git-based version control, CI/CD, automated testing, monitoring, documentation, deployment pipelines, change control and Infrastructure as Code.
6. Engineer for data quality, reliability and resilience implement validation, reconciliation, monitoring, audit, recovery, backfill and exception-handling processes.
7. Develop and optimise solutions across Microsoft Fabric, Snowflake and Azure contribute to performance, scalability, resilience and efficient resource utilisation.
8. Support advanced Analytics, Data Science and AI develop the pipelines, data structures and engineering components required for analytical and Machine Learning solutions.
9. Work collaboratively with analysts, Data Scientists and stakeholders understand downstream requirements and develop data products that solve genuine analytical and organisational problems.
10. Demonstrate the benefit of your engineering contribute to measurable improvements in processing time, manual intervention, reliability, data quality, reuse and analytical delivery.

Why join us?

You will work within an integrated Intelligence & Insights function supporting South East and South West London, alongside Data Platform Engineers, Data Scientists, Population Health specialists and wider analytical teams working with complex health and care data.Your engineering will directly support Population Health Management, strategic commissioning, planning, performance and wider analytical delivery. You will be able to see how your technical work translates into practical analytical capability and better decision support.You will also be joining during an important period of development, with the opportunity to work across Microsoft Fabric, Snowflake and Azure and to contribute to the evolution of modern Data Engineering practices across the function.If you are an experienced Data Platform Engineer who combines strong technical capability, curiosity, collaboration and a determination to turn good engineering into measurable analytical value, we would like to hear from you.

1. Design, build and optimise scalable data pipelines develop secure and reliable data flows from source systems through ingestion and transformation to curated analytical data products.
2. Develop data orchestration, transformation and integration solutions build and maintain automated ETL/ELT processes covering ingestion, cleansing, transformation, scheduling, dependency management and exception handling.
3. Build robust analytical data models translate analytical and business requirements into reusable dimensional, Lakehouse, Data Vault and semantic models.
4. Automate manual and repetitive processes identify opportunities to reduce unnecessary intervention and improve the productivity of Data Engineering and analytical teams.
5. Apply modern engineering disciplines use Git-based version control, CI/CD, automated testing, monitoring, documentation, deployment pipelines, change control and Infrastructure as Code.
6. Engineer for data quality, reliability and resilience implement validation, reconciliation, monitoring, audit, recovery, backfill and exception-handling processes.
7. Develop and optimise solutions across Microsoft Fabric, Snowflake and Azure contribute to performance, scalability, resilience and efficient resource utilisation
8. Support advanced Analytics, Data Science and AI develop the pipelines, data structures and engineering components required for analytical and Machine Learning solutions.
9. Work collaboratively with analysts, Data Scientists and stakeholders understand downstream requirements and develop data products that solve genuine analytical and organisational problems.
10. Demonstrate the benefit of your engineering contribute to measurable improvements in processing time, manual intervention, reliability, data quality, reuse and analytical delivery.

Please see attached job description for full details of the main responsbilities.

Person Specification

Qualifications

• Educated to degree level or equivalent level of experience working at a senior level in a specialist area, with additional in-depth professional knowledge and continuous professional development in own discipline
• Evidence of continued professional development

• Relevant certifications in Cloud & Platform systems, data architecture and design and related subjects

Experience needed - general or specific

• Extensive experience in Azure infrastructure design, deployment and management, ideally in an NHS setting
• Knowledge and experience of contributing and building complex ETL/ELT pipelines-airflow, data build tool, Fivetran, Matillion, Azure Data Factory
• Knowledge and experience of contributing to the integration of dozens of source systems
• Knowledge and experience of contributing to automated scheduling and orchestration
• Knowledge and experience for recovery and backfill processes
• Knowledge and experience of building reliable pipelines that handle large volume of data, near real-time feeds and complex transformations
• Knowledge and experience of performance optimisation
• Knowledge and experience of dimensional modelling, master data concepts, data lineage
• Knowledge and experience of data validation and reconciliation
• Knowledge and experience of GDPR, encryption
• Knowledge and experience of Masking and Tokenisation
• Knowledge and experience on developing machine learning pipelines
• Knowledge and experience of Machine learning Ops
• Demonstrable experience in maintaining an enterprise level Snowflake environment, including credit management, RBAC, and security governance, ideally in an NHS setting

• Knowledge and experience of Identity & Access Management, Role-based Access Control, and Attribute-based Access Control
• Knowledge and experience of agile product operating models of delivery

Skill and Ability

• The ability to perform system & architectural thinking as it relates to platform architecture, data storage, workload separation, stability design and resilience.
• The ability to undertake query tuning, warehousing sizing, resource monitoring and capacity planning.
• The ability to undertake data masking, implement row-level security, audit logging, data lineage.
• The ability to enable analytics and AI through the following: feature engineering pipelines, machine learning prep, MLOps, real-time data design
• The ability to identify, define, promote, communicate and achieve clear organisational values and goals, effective management processes, and rational and timely decision taking
• The ability to work in partnership with lead commissioners, clinicians and health care professionals within and outside the ICB and a partnership approach to working across organisations.
• The ability to analyse complex issues, to think and plan to achieve both tactical and strategic objectives, and to exercise sound judgement in the face of conflicting pressures.
• The skills to evaluate and learn from outcomes, with a clear commitment to innovation, learning and improvement.
• Problem solving skills and ability to prioritise and respond effectively to unplanned and conflicting demands.
• Autonomous strategic thinking abilities, to anticipate and mitigate against problems, formulating and adjusting plans as required, and briefing senior management whenever appropriate.
• Ability to provide and receive highly complex, sensitive, confidential, or contentious information to a range of technical and non-technical stakeholders, including senior management, negotiating and influencing them on difficult subjects.

• The ability to develop and maintain a data validation framework
• Ability to execute reconciliation methods, use monitoring tools and undertake incident management duties.
• Effectiveness in conflict resolution and management

What our service users expect of this post holder

• Stamina, energy and enthusiasm, and the ability to work effectively under pressure
• Robust knowledge of NHS IG principles and practical understanding of how to achieve compliance with legal requirements, particularly in relation to the processing of sensitive information

• Strong leadership influencing, negotiating and interpersonal skills
• A collaborative management style, which recognizes and values personal contributions and agenda and encourages team working
• Knowledge of current and evolving healthcare policies and digital strategies in London and nationally, and good understanding of their implications for South London

SNOWFLAKE

• Knowledge and experience of the three-layer architecture
• Knowledge and experience of Virtual warehouses, including multi-cluster warehouses, workload isolation
• Knowledge and experience of data ingestion and integration: bulk loading, continuous ingestion, connectors
• Knowledge and experience in data transformation & modelling: ELT patterns, dbt + Snowflake integration, stored procedures, star schema, building curate layer, incremental models, slowly changing dimensions
• Knowledge and experience of performance & cost optimization: clustering, micro-partitions, credit consumption, resource monitoring, warehouse sizing, query profile, account usage views, schema information
• Knowledge and experience of security, governance & compliance: access controls, data protection, compliance
• Knowledge and experience of infrastructure as code
• Knowledge and experience of automated deployments, zero-downtime change, disaster recovery

• Knowledge and experience of advance enterprise capabilities: data sharing, Snowpark, native apps, unistore and iceberg tables

MS Fabric

• Knowledge and experience of OneLake, Fabric workspaces and capacities
• Knowledge and experience of data factory: complex orchestration, parameterization, event-based triggers, retry and failure handling, meta-driven ingestion
• Knowledge and experience of Lakehouse: delta lake, table optimisation, ZS-order, travel time, schema evolution
• Knowledge and experience of Spark/Notebooks and warehousing
• Knowledge and experience onf DevOps: Fabric and Git integration, AzureDevOps/Gitub, branching strategy deployment pipelines and infrastructure as code
• Knowledge and experience in administration and configuration of GitHub and Azure DevOps code repositories for multi team use.
• Knowledge and experience of performance and cost engineering: query optimsation, file sizing, delta compaction, caching, capacity load patterns, capacity pooling.

• Knowledge and experience on how Fabric integrates with power BI semantic models, Directlake mode, DirectQuery
• Knowledge and experience of Identity & Access, data security, compliance
• Knowledge and experience of Kimball vs data Vault vs Lakehouse, dimensional modelling, slowly changing dimensions and even-driven pipelines

Other

• Experience and understanding of performance and change management, business planning, budget setting and monitoring, and contracting processes, in a health setting.
• Experience of managing the delivery and improvement of services within a complex and diverse organisation.
• Experience of supporting the development of strategies to meet changing organisational needs and acting as the technical subject matter expert across an entire directorate or Organisation
• Experience in designing, implementing and managing complex modern information and analytics systems, including Cloud-based data warehouses
• Experience in deploying, configuring, fine tuning, and managing MS Fabric environments and components

• Experience of leading teams, setting strategic and operational objectives, designing personal development plans, and resolving conflict situations
• Experience of inter-agency working and the development of joint projects, working with stakeholders to define success criteria, improving performance, and maximize efficiencies
• Experience in contributing to the planning of complex technical projects, such as the adoption of innovative solutions and the migration across different analytics systems
• Experience in liaising with ICT, IG, cyber security and other related disciplines and internal and external teams to ensure their timely cooperation on projects that require their input or sign off
• Extensive experience in reviewing and amending large volumes of complex code and identifying performance issues and/or bugs checking for adherence to coding best practice and local standards
• Experience in working under pressure and in uncertain and highly politically sensitive environments

Disclosure and Barring Service Check

This post is subject to the Rehabilitation of Offenders Act (Exceptions Order) 1975 and as such it will be necessary for a submission for Disclosure to be made to the Disclosure and Barring Service (formerly known as CRB) to check for any previous criminal convictions.

Employer details

Employer name

South West London Integrated Care Board

Address

Tooley Street/120 The Broadway

London

SE1 2TZ

United Kingdom

Employer's website

https://www.southwestlondon.icb.nhs.uk