SC Cleared Senior Data Engineer (SpaceObserver Implementation) - Remote - Inside IR35

About Us:

Solirius Reply, part of the Reply Group, is a technology consultancy and digital transformation partner that helps organisations solve complex challenges through strategy, design, engineering, and delivery.

We work closely with our clients to deliver secure, accessible, user-focused services that evolve with their needs. By combining deep technical expertise with people-centred design, we create solutions that deliver meaningful, lasting impact.

Our consultants partner directly with client teams, embedding into organisations to understand their goals, challenges, and users. This collaborative approach enables us to deliver tailored solutions that drive measurable outcomes across public and private sectors.

Past and present clients include the Ministry of Justice, Department for Education, Ministry of Housing, Communities and Local Government, UEFA, International Olympic Committee, and Mercedes-Benz. Our services span the full digital delivery life cycle, including architecture, engineering, delivery management, user-centred design, business analysis, data, DevOps, and AI.

We operate as a collaborative and inclusive organisation that empowers our people to take ownership, innovate, and develop their expertise. As an equal opportunities employer, we are committed to encouraging equality, diversity, and social mobility, while creating opportunities for our teams to work on meaningful projects that deliver lasting impact.

About You:

You are a motivated and adaptable professional with a strong analytical mindset and a passion for using technology to solve real-world problems. You enjoy working in collaborative, agile teams and take pride in delivering high-quality solutions that make a tangible impact. With strong communication skills and a consultative approach, you're comfortable engaging with clients, understanding their needs, and translating them into effective outcomes.

The Role:

Our client requires a highly structured, hands-on Senior Data Engineer to execute the setup, configuration, and data processing paths for the SpaceObServer Enterprise software tool and its Web Access add-ons.

The department's operational estate data is unstructured and scattered across thousands of disparate SharePoint site collections. Recent data discovery sweeps indicate an active environment of 71.6M to 75.3M files; consequently, you will be architecting and automating an implementation provisioned for a target baseline of 80 million files to mitigate system latency and risk.

Operating strictly within pre-approved Technical Design Authority (TDA) guidelines, you will act as the technical engine translating approved architectural designs into production-ready cloud components, automation scripts, and optimised scanning paths.

Key Responsibilities

1. Automated IaC Engineering

• Develop, test, and establish robust, version-controlled Infrastructure-as-Code (IaC) templates (primarily Terraform blueprints and Azure DevOps pipelines).

• Automate the deployment and hardening of the target Azure infrastructure environment supporting the SpaceObServer scanning engine and its backing SQL database instance (Azure SQL/SQL Managed Instance).

2. Programmatic Identity Configuration & Rate-Limit Optimisation

• Script and validate the setup of a tenant-wide Microsoft Entra ID Application Registration pool array (5 to 10 applications).

• Configure application-level load balancing and rotation scripts to programmatically distribute scanning traffic, explicitly designed to handle deep Microsoft Graph API and SharePoint REST API rate limits/throttling over 80M+ items.

3. SpaceObServer Configuration & Data Processing

• Translate pre-approved TDA architectures into active SpaceObServer configuration files.

• Establish optimal background scanning schedules, indexing priorities, and target processing paths across thousands of unstructured SharePoint site collections.

• Integrate and configure the SpaceObServer Web Access add-on for downstream departmental reporting.

Technical Skills & Experience Required

• Enterprise Tool Implementation: Proven experience configuring and deploying enterprise-scale data discovery, storage analysis, or indexing tools (direct experience with SpaceObServer Enterprise or equivalent tools like TreeSize/JAM Software suites is highly advantageous).

• Scale Competency: Proven track record of handling massive data discovery/migration pipelines without blowing out latency or failing system API thresholds.

• Advanced Azure DevOps & Terraform: Deep expertise building complex Azure infrastructure through automated CI/CD pipelines and modular Terraform configurations.

• Microsoft Graph/SharePoint REST API: Expert-level understanding of Microsoft Entra ID (Azure AD), app registrations, multi-factor authentication integration via certificates, and methods used to bypass or mitigate Microsoft Graph API throttling under heavy loads.

• Experience delivering within the public sector under GDS Beta phase guidelines.

• Prior experience working strictly as an implementation partner interfacing with a rigid Technical Design Authority (TDA).