Big Data Engineer

Big Data Engineer

£80,000 - £85,000 + Bonus + Strong Benefits

Hybrid – South West / Remote Flexibility

SC Eligible Required (Must be capable of obtaining DV Clearance)

We’re supporting a highly technical research-led organisation working on genuinely complex large-scale data problems within a secure environment.

This is not a traditional “move data from A to B” Data Engineering role.

The focus is on designing and building distributed data processing systems capable of ingesting, transforming and analysing terabytes of largely textual and semi-structured data at significant scale. The work involves extracting meaning and relationships from fragmented datasets, helping build advanced knowledge and intelligence-driven platforms.

The environment is heavily engineering-focused and would suit someone who enjoys solving difficult technical challenges around scalability, distributed systems, unstructured data processing, streaming architectures and high-volume data ingestion.

The Role

You’ll be joining a specialist engineering team focused on large-scale data processing and analytics platforms.

The work will involve:

• Designing scalable distributed data pipelines for large textual and semi-structured datasets
• Processing and transforming terabytes of high-volume data into structured and queryable formats
• Building ingestion and transformation frameworks for complex multi-source datasets
• Working with streaming and event-driven architectures
• Supporting graph, entity-resolution or relationship-driven data models
• Optimising data processing performance across large distributed environments
• Collaborating with highly technical researchers, engineers and data specialists
• Helping shape long-term platform architecture and scalability decisions

This is a role for someone who enjoys deep technical engineering rather than pure reporting, dashboarding or standard ETL delivery.

Key Experience

We’re interested in engineers who have worked on genuinely large-scale or technically challenging data environments.

Strong experience in several of the following would be beneficial:

• Big Data Engineering
• Distributed Systems
• High-volume textual or unstructured data processing
• Apache Spark / PySpark / Scala
• Kafka or streaming/event-driven architectures
• Elasticsearch / OpenSearch
• Knowledge Graphs / Entity Resolution / NLP / Search platforms
• Data modelling for large-scale analytics systems
• Kubernetes / Docker / Terraform
• AWS-based data platforms
• Real-time or near real-time processing pipelines
• Data lineage, governance or metadata-heavy environments

Backgrounds of interest could include:

• Financial crime / fraud analytics
• Intelligence or research environments
• Scientific or genomics platforms
• Search/recommendation engines
• Large-scale digital platforms
• Complex cloud-native data ecosystems

Security Requirements

Candidates must be capable of obtaining DV Clearance.

Due to the nature of the work, preference will be given to:

• Sole UK Nationals
• Candidates with long-term UK residency
• Individuals eligible for SC clearance immediately

Existing SC or DV clearance would be highly advantageous.

Package

• £80,000 - £85,000 base salary
• Bonus 10%
• Strong pension
• Flexible hybrid working
• Highly technical engineering environment
• Long-term complex project work
• Excellent learning and progression opportunities

If you enjoy solving genuinely difficult data engineering problems at scale and want to work in an environment focused on advanced distributed systems and large-scale data processing, I’d be happy to discuss the opportunity further.