Search papers, labs, and topics across Lattice.
The authors validated LLM-based agent judges against human raters across 15 tasks, finding indistinguishable performance in a Turing-style test. They then characterized the scaling behavior of these judges, observing that quality scores improve logarithmically with panel size while unique issue discovery follows a sublinear power law. This score-coverage dissociation is attributed to a power law distribution of the finding space, with structured persona conditioning enhancing ensemble diversity and accelerating issue discovery.
LLM-based agent judges hit diminishing returns surprisingly fast: quality scores saturate twice as quickly as unique issue discovery, revealing a fundamental disconnect between perceived quality and comprehensive coverage.
LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score-coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law-both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity-Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties.
