Engineering AI Engineer (Harness) London — Full time Apply →
- Location
- Greater London, England, United Kingdom
agreement metrics (Cohen’s kappa, correlation, precision/recall) and systematically detect judge failure modes such as position bias, verbosity bias, self-preference, and prompt sensitivity. Curate Benchmark Datasets & Rubrics: Define sampling strategies, detailed annotation guidelines, and scoring rubrics. Measure inter-annotator agreement and manage label noise to ensure … Error & Trajectory Analysis: Conduct hands‐on failure analysis on agent runs, cluster root causes into actionable error categories, and clearly communicate findings to both engineering teams and stakeholders. Drive Harness Scaffolding Iteration: Translate evaluation results directly into architectural improvements across agent scaffolding, prompt design, tool execution loops, context ...