Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper formalizes the concept of a claim-replay layer in evaluation artifacts, highlighting the limitations of current metrics that do not adequately license the claims associated with them. By conducting a census of 124 Inspect Evals units, the authors find that 110 of these units fail to reach deterministic inference due to a lack of necessary historical evidence or semantic grounding. The resulting audit reveals a nuanced landscape of evaluation stability and claim resolution, emphasizing the need for a more rigorous framework in assessing evaluation artifacts.
110 out of 124 evaluation units fail to provide deterministic inference, exposing critical gaps in how claims are validated in AI assessments.
Evaluation artifacts specify a forward computation: a task, scorer, and reported metric. They do not necessarily license the claim attached to that metric because the historical evidence and alternative semantics needed to replay it may be unbound. We formalize this missing claim-replay layer through a frozen substrate D, a grounded family F, a claim query q, and the resulting identified set. We then census all 124 mechanically eligible Inspect Evals units at a pinned commit. Every unit receives a terminal disposition; 110 stop before deterministic inference because required historical evidence or semantic grounding is unavailable. Where execution closes, exact values, winners, complete orders, and pairwise relations separate by claim resolution and by primary versus review family. The audit therefore returns typed stops, instability witnesses, and stable substructure rather than forcing one evaluator meaning or one robust/not-robust label.