Search papers, labs, and topics across Lattice.
The paper introduces Seeded Elimination with Adaptive LLM-as-a-Meta-Judge (SEAL), a novel evaluation protocol designed to revive saturated LLM benchmarks by extracting ranking signals from existing task outputs. SEAL employs a self-improving LLM judge within an elimination tournament framework, using task-specific principles and adaptive checklist criteria to evaluate candidate outputs. Experiments across code generation, mathematical reasoning, knowledge QA, and tool-use demonstrate that SEAL achieves high Spearman correlation (0.83-1.00) with full pairwise judging while significantly reducing the number of LLM calls (11.89 vs 28.00).
Saturated LLM benchmarks can be revived without creating new datasets: a self-improving LLM judge in an elimination tournament recovers ranking signal and breaks ties.
Widely used language-model benchmarks are increasingly saturated, with frontier systems often receiving near-tied scores that standard metrics cannot resolve. Rather than constructing harder alternatives, we ask whether existing tasks can be made informative again through improved evaluation over the same candidate outputs. Therefore, we present Seeded Elimination with Adaptive LLM-as-a-Meta-Judge, a self-improving evaluation protocol for extracting latent ranking signal from saturated benchmarks. SEAL seeds candidate outputs into a single elimination and evaluates each match with task-level principles plus self-improving checklist criteria. We evaluate SEAL on multiple saturated benchmarks covering code generation, mathematical reasoning, knowledge-intensive question answering, and tool-use agent task completion. Across these settings, SEAL improves the ranking-accuracy--latency trade-off over competing protocols, attaining 0.83--1.00 Spearman agreement with full pairwise judging and 4/4 top-1 agreement, while requiring only 11.89 calls per task compared with 28.00 for full pairwise evaluation.
