Lead Data Engineer

Lead Data Engineer

London (Hybrid, 1+ day per week on-site)

A standout opportunity for a senior engineer who wants to lead a platform team while staying hands-on and technical.

We're partnering with a well-backed B2B SaaS business whose software underpins millions of transactions a year. Several established products now sit under one platform, and the company is putting serious investment behind the data infrastructure that ties it all together. It's a business with genuine scale and stability, but the pace and ambition of something far younger, and it's going through a period of real growth and change.

The brief is to lead a small team of platform engineers who build the foundation the rest of the data organisation depends on. This is a true 50/50 role: half hands-on engineering, half people and delivery leadership. Nobody here is stepping away from the technical work.

The Role

You'll report into the Head of Platform & Data Engineering and work hand in glove with the Technical Product Manager, taking day-to-day ownership of around three engineers. The mission is to scale the Databricks lakehouse, stand up the infrastructure behind new data commercialisation products, and create the reusable components that let teams across the business work with data quickly and safely. The platform is already on a modern footing, so the emphasis is squarely on what comes next: extending what it can do, tightening reliability and performance, and shipping the tooling that helps everyone else move faster.

What you'll be doing

• Splitting your time roughly evenly between building and leading, writing Python and Terraform, designing systems and reviewing code alongside your team
• Growing and developing the engineers around you through coaching, clear objectives and honest, regular feedback
• Creating the templates, abstractions and paved roads that make other engineers more productive, and getting your team thinking the same way
• Owning features end to end, from first design through to production
• Working with the Technical Product Manager to set the roadmap and make sensible calls when priorities compete
• Keeping the platform resilient, performant and ready for whatever's next
• Partnering with product, commercial and engineering colleagues to turn messy business problems into clean, customer-facing data solutions

What we're looking for

Essential

• Strong Python and a track record of building data platform infrastructure that other engineers and analysts depend on
• Deep, hands-on AWS and Terraform, with infrastructure as code as your default way of working
• Experience building on a modern lakehouse or warehouse (Databricks ideally, though Snowflake or BigQuery travels well)
• Ingestion and transformation tooling such as Fivetran and dbt
• Experience managing software engineers, ideally in a data or platform setting, across hiring, performance and career growth
• A genuine commitment to staying technical, seeing hands-on engineering as central to the role rather than something you've grown out of

Nice to have

• Working knowledge of streaming systems such as Kafka, and a feel for the abstractions that make event-driven data easy for other teams to consume
• Solid SQL and a good sense for data modelling, even if neither is your daily focus
• Awareness of how data workloads run in production on ECS and Kubernetes

The Stack

Core

• Python and SQL
• Databricks, with lakehouse storage on S3
• AWS (EventBridge, Kinesis, Lambda, S3, EC2) with Terraform for infrastructure as code

Supporting

• dbt and Fivetran for transformation and ingestion
• Kafka for streaming
• ECS and Kubernetes for orchestration

If this sounds like you and you'd like to hear more (or explore other roles), apply here or reach me directly at niall.wharton@Xcede.com. Feel free to attach a CV for review.