Lead Data Engineer

We’re looking for Lead or Principle Data Engineers to help build and evolve a secure batch and real‐time data platform in a cloud environment. You’ll work on ingesting data from multiple sources, transforming it into trusted formats, and making it available for downstream users and analytics.

You’ll be working on secure data platforms within the national security sector, supporting mission‐critical systems.

What you’ll be doing

• Designing, building and maintaining data pipelines
• Developing batch and real‐time data processing solutions
• Performing data modelling and database design
• Analysing data issues and identifying improvements
• Ensuring data aligns to agreed target models and architecture
• Supporting and improving a growing data platform

What we’re looking for

• Experience building data pipelines and data processing solutions
• Strong understanding of data modelling and data structures
• Hands‐on experience with AWS (EC2, RDS, Lambda, Athena, Glue desirable)
• Python or similar scripting/programming experience
• Experience with relational and NoSQL databases
• Familiarity with Agile delivery and Test‐Driven Development

Nice to have

• CI/CD tools such as Jenkins or Drone
• Infrastructure‐as‐Code (Terraform)
• Configuration tools like Ansible or Packer
• Linux and Windows environments