Search papers, labs, and topics across Lattice.
The paper introduces Interactive Benchmarks, a new evaluation paradigm for assessing reasoning abilities in LLMs through interactive problem-solving under budget constraints. This paradigm is instantiated in two settings: Interactive Proofs (logic/math) and Interactive Games (strategic reasoning). Experiments demonstrate that Interactive Benchmarks offer a more robust and faithful assessment of model intelligence compared to standard benchmarks, highlighting significant areas for improvement in interactive scenarios.
LLMs still have a long way to go when it comes to interactive problem-solving, as revealed by a new benchmark that tests reasoning under budget constraints.
Standard benchmarks have become increasingly unreliable due to saturation, subjectivity, and poor generalization. We argue that evaluating model's ability to acquire information actively is important to assess model's intelligence. We propose Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints. We instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities. Our results show that interactive benchmarks provide a robust and faithful assessment of model intelligence, revealing that there is still substantial room to improve in interactive scenarios. Project page: https://github.com/interactivebench/interactivebench
