Search papers, labs, and topics across Lattice.
This paper analyzes public AI evaluation archives, revealing that the reported performance metrics are influenced by selective reporting and benchmark changes, leading to potential misinterpretations of AI capabilities. By employing Bayesian inference, the authors demonstrate that two distinct historical paths can yield similar terminal outcomes, complicating the understanding of AI progress. The study introduces an archive-and-adjudication protocol that effectively reconstructs evaluation histories and identifies unsupported claims, enhancing the reliability of public AI assessments.
Misleading AI performance metrics can stem from selective reporting, with two different histories yielding similar results, challenging our understanding of progress in AI capabilities.
Public AI evaluations are often read as terminal leaderboards, yet the underlying evidence is a selective time series shaped by reporting rules, benchmark revisions, and missingness. Repeated public archives for LiveBench and Open LLM Leaderboard v2 serve as the primary longitudinal record; LMArena provides a preference stress test; and GAIA and tau-bench contribute limited agentic pilots. Together, these archives instantiate a Bayesian inference problem: under a fixed reporting convention, one constructed terminal-only example over $1{,}000$ systems is compatible with two pre-terminal histories, yielding times of $23.03$ or $75.13$ to reach within $0.05$ of the ceiling under the same terminal-tail model. In synthetic posterior comparisons, action-facing diagnostics differ across observation regimes. The candidate selection-aware frontier model fails synthetic recovery, objective-archive prediction, preference transfer, and uncertainty calibration; correspondingly, fixed audit gates reject its stronger claims. An archive-and-adjudication protocol reconstructs public evaluation histories, isolates a verified timing boundary, and falsifies unsupported frontier claims.
