Search papers, labs, and topics across Lattice.
This study investigates how varying the token generation budget affects the evaluation of large language models (LLMs) across different reasoning tasks. The authors found that model rankings can reverse depending on the budget, with 3-19% of items showing non-monotone behavior, indicating that traditional evaluation methods may overlook significant model-specific performance variations. Additionally, a budget-aware routing approach was developed that captures a notable portion of the oracle gap, highlighting the importance of budget considerations in LLM evaluations.
Model rankings can flip dramatically based on token generation budgets, revealing hidden performance dynamics that challenge standard evaluation practices.
Standard evaluation of large language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels (64--4,096), evaluating four models on three reasoning benchmarks (56,476 inferences). We report four findings: (i) 3--19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6--14%). (ii) Model rankings reverse across budgets on all benchmarks ($p {<} 0.01$, McNemar). (iii) Oracle analysis reveals model complementarity up to $+27.8$pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain ($+1.6$ to $+5.7$pp) but are domain-specific and hurt transfer ($-1.2$pp). These results argue for budget-conditioned evaluation protocols.
