Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critiques the common practice of extrapolating AI benchmark results to broader claims about system capabilities, highlighting the epistemic issues that arise when such inferences are made without sufficient evidence. It introduces the concept of projectibility, emphasizing that warranted links between benchmarks and their applications do not guarantee a valid chain of reasoning, particularly when conditions or assumptions differ. Through a legal-research case study and simulations, the authors demonstrate how seemingly stable aggregate results can obscure critical distinctions necessary for valid projections, ultimately proposing a projectibility audit to identify unsupported claims in benchmark evaluations.
Benchmark evaluations can mislead when their results are inappropriately extrapolated, risking unsupported claims about AI capabilities.
An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paper's distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.