Lead Observability & Data Architect/Engineer (SC/DV Cleared)

We are hiring for Lead Observability & Data Architect/Engineer (SC/DV Cleared)

Location: Hybrid - London/Reading/Birmingham/Manchester/Glasgow

• We are seeking an Enterprise Platform Architect or Senior Data Engineer to design and scale national-critical monitoring fabrics and data hubs. Whether your background is rooted in core Observability platforms (Opensearch/Elastic/Splunk/OpenTelemetry) or high-scale Data Engineering (Kafka, Python, Airflow, OpenSearch/Lucene), your core mission is the same: transforming billions of raw event logs into actionable, secure streaming telemetry.
• Should be able to build high-performance data pipelines to ingest, parse, and structure massive event telemetry across hybrid-cloud and air-gapped environments.
• Manage large-scale cluster deployments, designing optimized indexing, data life cycle policies, and schema mapping (such as ECS).
• Expertise in Elasticsearch/OpenSearch, Splunk, or Solr. Strong proficiency in building visualization dashboards (Kibana, Grafana) and managing telemetry collectors (OpenTelemetry/OTel, Beats, Prometheus).
• Experience in designing and scaling data pipelines for high-velocity telemetry. Essential expertise in Apache Kafka, Logstash, or streaming/orchestration frameworks (Airflow, Vector, Debezium).
• Hands-on experience architecting scalable, containerized solutions using Kubernetes (K8s), Docker, and infrastructure-as-code (Terraform/Ansible).
• Search & Indexing Engine Depth: Prior architecture experience with OpenSearch, Apache Solr, or Lucene core or Elasticsearch.
• Strong proficiency in Python, Rust, or Bash for automating log parsing, testing, and deployment workflows.
• A strong track record of engineering within highly regulated sectors (eg, MoD, Home Office, BAE, HMRC) with an active UK security clearance (SC/DV).

We highly encourage applications if you bring any of the following equivalent expertise:

• From Big Data/ETL to Telemetry Pipelines: If you have built high-throughput streaming pipelines using Apache Kafka, Airflow, NiFi, or Spark, you already know how to route and transform complex data feeds at scale.
• From Enterprise Search to System Observability: If you have engineered core search, indexing, or sharding architectures using Apache Solr, Lucene, or OpenSearch, you possess the exact foundational knowledge needed to scale an enterprise Elasticsearch cluster.
• From Cloud Infrastructure to Monitoring Fabrics: If you are a seasoned Cloud or Infrastructure Architect specializing in secure Kubernetes orchestration, Terraform automation, and air-gapped system design, you can easily pivot to defining secure logging architectures.