Search papers, labs, and topics across Lattice.
This study investigates how varying inference compute influences the evaluation of frontier language models across seven challenging benchmarks in diverse domains such as software engineering and cybersecurity. By implementing three inference-scaling interventions鈥攍arger token budgets, context compaction, and repeated submissions鈥攖he authors reveal that performance improvements are substantial when models are given more resources, with fixed-budget evaluations often misrepresenting true model capabilities. The findings highlight that different benchmarks respond variably to these interventions, underscoring the need for nuanced evaluation protocols that reflect the actual capabilities of advanced models under varying compute conditions.
Fixed-budget evaluations can significantly underestimate the capabilities of advanced language models, with larger token budgets unlocking their true potential across diverse tasks.
AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving. As a result, performance is increasingly sensitive to the amount and allocation of compute available at test time ("inference compute"). Yet many evaluations still report performance at a single restrictive budget, meaning that low scores may reflect the evaluation setup rather than the model's underlying capability. To test this, we evaluate up to 12 frontier language models on seven challenging benchmarks spanning software engineering, mathematics, medicine, and cybersecurity. We use a controlled setup combining three simple inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts, guided either by the model itself or by minimal correctness feedback. We find three main results. First, larger token budgets substantially improve performance on benchmarks across multiple domains, including cybersecurity, FrontierMath, Humanity's Last Exam, and TerminalBench. Second, fixed-budget evaluations can increasingly understate frontier capability as models advance. Newer models reach higher performance at large budgets, where they unlock harder tasks and solve them more reliably. Third, benchmarks differ in which inference-scaling methods help most: repeated submission broadly improves performance, but the value of larger token budgets, external feedback, and parallel attempts varies by benchmark. Overall, our results show that benchmark scores are protocol-dependent. We therefore argue that evaluations should report capability as a function of inference-time compute, specify protocol choices explicitly, and compare model generations over a large shared compute range at matched budgets, especially in safety- or policy-relevant settings.
