Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critiques the conventional accuracy metrics used to evaluate language models, revealing that they obscure critical failure states by conflating execution evidence with correctness judgments. By employing a two-layer evaluation framework, the authors analyze 2,550 outputs across various model configurations, uncovering significant discrepancies in termination and answer exposure rates under matched output budgets. The findings highlight that accuracy alone can misrepresent model performance, necessitating a more nuanced reporting of execution states and verification processes in evaluations.
Accuracy metrics can mask critical failure states in language models, revealing that nearly 11% of outputs can terminate without a final answer under specific conditions.
Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.