AI Product Analyst (AI Metrics & Model Evaluation)

AI Product Analyst (AI Metrics & Model Evaluation) | London | £80,000-£85,000

Join an award-winning, internationally recognised consultancy that's growing fast and building client-facing, cutting-edge AI products used across highly regulated industries, where real users act on high-level advice relating to legislative, regulatory and legal frameworks.

As AI Product Analyst you'll play a key role in providing high-level insight into how much the AI products are being used, how well the AI models and agents are working, and reporting on the key analytics - segmenting the data to highlight trends by client, sector and question type. You will own the AI metrics and model evaluation capability that shapes the next stage of generative and agentic AI development.

The role splits into two dynamic elements: AI metrics tell you how the product is performing in the hands of actual users, and model evaluation tells you whether a change is good enough to release. You'll report into the AI leadership team and work day to day with Product, Engineering and the in-house experts who know what a correct answer looks like. The scope will include;

• Defining the AI metrics that matter, retrieval quality, correctness of output, and what users go on to do with what they're given
• Owning production quality observability, from thumbs-down and regeneration rates through to abandoned tasks and drift in how the product is being used
• Building the analysis and dashboards the team works from, and specifying what needs capturing when the product itself has to change to get the AI metrics you need
• Digging into the data to work out what's really going on, the patterns, the outliers and the likely causes, then deciding which of it matters commercially
• Choosing the right method for the question, whether that's segmentation, cohort comparison, trend analysis or something built for the purpose, rather than reporting the same six metrics every month
• Owning the model evaluation programme: what gets tested, how much it covers, and what the results actually tell us
• Owning the golden datasets that underpin model evaluation, building them, extending them, versioning them and keeping them honest and running elicitation sessions with subject-matter experts to capture the professional judgement behind a right answer
• Turning findings into recommendations and following them through, so the insight changes something rather than sitting in a report
• Saying so clearly when model evaluation coverage isn't strong enough to call a release

Essential Experience

• Solid experience as a product, data or commercial analyst, having owned measurement for a product or service end to end
• Strong hands-on SQL and Python, to the point where you build your own analysis and your own dashboards rather than queuing for someone else to do it
• The judgement to choose the right method for a question, and to know which lines of enquiry are worth your time
• Comfort working with the output of AI or machine-learning systems, and a clear view of why AI metrics and model evaluation differ from measuring a deterministic product
• A track record of getting knowledge out of experts and turning it into something structured and reusable
• The ability to put a recommendation in front of senior people and hold your position, particularly when they don't want to hear it

Desirable experience

• Hands-on model evaluation with LLMs: golden datasets, scoring and regression testing.
• Retrieval-augmented systems and the AI metrics used to assess them
• Experiment design and A/B testing
• A regulated professional services environment

This is a defining appointment for a fast-growing AI function, and a rare opportunity to build an AI metrics and model evaluation capability from first principles rather than inherit one. You will set the standards against which product quality is judged and release decisions are made, so your work determines what ships, not what gets reported afterwards. The role carries genuine autonomy, direct exposure to senior leadership, and the backing of an executive team with a clear AI roadmap.

INDAMS

The Portfolio Group are acting on behalf of our client in recruiting for this position.