Senior Data Platform Engineer

Senior Data Engineer – Platform & APIs or Senior Data Engineer – Data Exposure

Intelligence Data Platform — Manchester (or Croydon)

What You'll Own

End-to-End Data Ownership

• Raw data ingestion through final consumption
• Architecture decisions that affect downstream users
• Trade-offs between speed, accuracy, and cost
• You're accountable for the whole chain, not a single layer

API Design & Delivery

• Build REST APIs that serve data to multiple downstream teams
• Design endpoints that reflect data structure and user needs
• Manage API performance, versioning, and reliability
• Own the contract between data and consumers

Data Pipeline Architecture

• Design and optimise ingestion, transformation, and delivery systems
• Work with complex, relationship-driven datasets (knowledge graphs, entity relationships)
• Structure data for both query performance and downstream usability
• Handle schema evolution as data structures change

Quality & Observability

• Build validation, testing, and monitoring into systems from day one
• Design for failure recovery and data consistency
• Prevent issues before they happen, not react to them after
• Understand how systems degrade and what breaks first

Performance & Reliability

• Optimise query performance on 200M+ node datasets
• Make informed decisions about where to optimise and where to accept trade-offs
• Design systems that scale as data volume grows
• Collaborate with architects on bottleneck resolution

Collaboration

• Work closely with architects and platform teams on technical decisions
• Understand how downstream teams consume your APIs
• Communicate trade-offs and constraints clearly

What You're NOT Doing

• You're not an ETL specialist. You own the full stack, not just pipelines.
• You're not handing off to someone else to expose your data. You build the APIs.
• You're not working with clean, well-structured data. You deal with complexity.
• You're not reactive about quality. You build reliability in from the start.
• You're not a single-layer expert. You understand how all layers interact.

You Must Have

Built REST APIs or data-serving endpoints (non-negotiable)

• You've designed and built APIs that other teams depend on
• You understand REST conventions, versioning, error handling, documentation
• You've made design decisions about payloads, endpoints, performance
• Examples: Flask/FastAPI services, Node.js endpoints, GraphQL resolvers, or similar
• Not: Consuming APIs via Azure Data Factory linked services. Not BI dashboards. Not streaming ETL outputs to tables.

Owned data end-to-end

• Raw ingestion through to how users consume the final output
• Not a specialist in one layer (ingestion-only, ETL-only, warehouse-only)
• You've made decisions about how to structure data for both performance and usability
• You understand the impact of your schema design on downstream queries

Worked with complex, structured data at scale

• Relationship-driven datasets where structure matters
• Entity modeling, dimensional architecture, or knowledge structures
• TBs of data, millions of records, systems where schema design affects everything
• Not simple flat data or reporting-focused analytics

Quality-first mindset

• Testing, validation, and monitoring built into systems from day one
• Tools like dbt, Great Expectations, schema validation aren't novel to you
• You think proactively about failure modes, not reactively after incidents
• You understand data quality as a systems problem, not a cleanup task

Cloud platform experience

• AWS preferred (hands-on with compute, storage, networking services)
• GCP or Azure acceptable if you understand cloud patterns and can transfer to AWS
• You've made infrastructure decisions, not just executed someone else's design

Nice to Have

• Neo4j or graph database experience (knowledge graphs are core to what we do)
• Event-driven architecture or streaming systems (Kafka, Kinesis)
• Databricks, Snowflake, or modern data stack experience
• Data warehouse or lakehouse design
• Infrastructure as Code, CI/CD, observability tooling
• Python or JavaScript backend frameworks

Why This Matters

This organisation sits at the intersection of data, intelligence, and critical infrastructure. We serve governments, defence organisations, and commercial enterprises with structured, verified intelligence.

The systems you build directly impact how decision-makers access critical information. You're not doing routine analytics work. You're solving genuine technical challenges:

• Optimising query performance on 200M+ node knowledge graphs
• Designing schema evolution strategies for complex, multi-source data
• Building APIs that serve diverse downstream teams reliably
• Making architectural trade-offs between accuracy, speed, and cost
• Handling relationship-driven data where structure is everything

Working Setup

• Hybrid: 2 days per week in Manchester or Croydon office
• Technically-focused team with architects you'll work closely with
• Opportunity to shape technical direction and engineering standards
• Real technical challenges, not routine delivery
• Access to complex, interesting data problems

About You

You're a full-stack data engineer who sees data as a systems problem. You don't think in layers (ETL, warehouse, BI)—you think in data flows. You've built things that other teams depend on, including APIs. You care about how downstream users work with your data. You make trade-off decisions thoughtfully. You build quality in rather than patching it later.

You're motivated by solving real technical problems at scale, not by tooling fads. You understand that good data architecture affects everything downstream, and you take that responsibility seriously.