Search papers, labs, and topics across Lattice.
This paper introduces Active-SWE, a new benchmark designed to evaluate coding agents in proactively discovering and fixing bugs without relying on issue reports, addressing a significant gap in existing software engineering benchmarks. The benchmark encompasses 1,663 tasks across six bug categories and eight programming languages, shifting the focus from reactive to proactive bug fixing. Experimental results show that current state-of-the-art coding agents exhibit limited capabilities in these proactive tasks, highlighting the challenges in locating and resolving multiple bugs effectively.
Most coding agents fail to proactively fix bugs without issue reports, revealing a critical gap in their capabilities.
Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase. However, existing SWE benchmarks typically assume that high-quality issue reports with detailed information are always available, which is easily violated in practice due to the complexity of report acquisition and curation. To address this, we introduce Active-SWE, a benchmark for evaluating coding agents on proactively discovering and fixing multiple bugs without report guidance, covering 1,663 tasks across six bug categories and eight languages. Beyond shifting the focus from existing reactive bug fixing to proactive bug fixing, Active-SWE enables a more in-depth evaluation by expanding the scope from fixing a specific recorded bug to multiple-bug fixing and potential bug discovery scenarios. To construct Active-SWE, we propose a novel difficulty-aware task formulation pipeline with a dual-track evaluation framework, facilitating comprehensive evaluation of proactive bug-fixing capability. Extensive experiments reveal that most state-of-the-art coding agents struggle with proactive bug-fixing tasks, demonstrating limited performance in locating and resolving recorded bugs, handling multiple bug fixing scenarios, and discovering valid potential bugs.
