Search papers, labs, and topics across Lattice.
This paper introduces InteractBench, a novel benchmark designed to evaluate large language models (LLMs) on competitive programming tasks that require dynamic interaction and information acquisition under constrained conditions. By focusing on interactive problems, which necessitate multi-round engagement with an interactor and limited query budgets, the authors highlight a critical gap in existing benchmarks that typically assess full-information scenarios. The evaluation of LLMs on this benchmark reveals a significant interaction gap, with even state-of-the-art models struggling to perform effectively, underscoring the need for improved algorithmic reasoning capabilities in dynamic contexts.
LLMs struggle with interactive programming tasks, revealing a significant gap in their algorithmic reasoning abilities under constrained information conditions.
Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent. Code is available at https://github.com/kmsgk0/InteractBench.
