Search papers, labs, and topics across Lattice.
This study introduces the Capability Frontier, a novel evaluation framework that quantifies the performance of multiple LLMs under heterogeneous data distributions, revealing that traditional benchmarks significantly underestimate model capabilities. By analyzing 21 LLMs across 16 benchmarks, the authors demonstrate that correcting for single-model evaluations leads to a 54% reduction in error rates, while accounting for single runs results in an 82% improvement in perceived performance. The findings indicate that collective capabilities of LLMs are vastly underestimated, with potential implications for their evaluation and deployment in diverse real-world scenarios.
Traditional benchmarks miss 82% of LLM performance, revealing a vast underestimation of their true capabilities in diverse tasks.
Existing benchmarks typically report accuracy for a single model on a single run. This systematically understates real-world LLM capabilities, particularly under heterogeneous data distributions: (i) different models get different questions correct according to their specializations, and (ii) given a budget, multiple generations can be sampled and selectively retained. To quantify this gap, we introduce the Capability Frontier: a Pareto frontier over a set of models that characterizes the best achievable performance at each cost level under optimal selection across models and generations (i.e., via an oracle). Our construction corrects for two opposing biases: underestimation from single-model evaluation and overestimation from taking maxima over noisy samples. We study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks, comparing Capability Frontier performance at matched cost to each benchmark's top-performing model. Correcting for single-model evaluation yields a 54% error rate reduction; additionally correcting for single runs yields an 82% improvement, with SOTA accuracy matched at 85% cost reduction. Complementing these empirical results, we use controlled probabilistic simulations to show that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model. Our findings suggest collective LLM capabilities are substantially underestimated, with implications for evaluation and deployment in data-heterogeneous, multi-domain settings.
