Search papers, labs, and topics across Lattice.
This paper introduces SWE-Gate, a novel benchmark that evaluates software engineering agents not only on their ability to pass functional tests but also on their compliance with review-derived acceptance constraints. By analyzing real pull request review comments, SWE-Gate synthesizes 303 repository-level repair instances from 75 open-source Python repositories, providing a comprehensive assessment of coding agents' performance. The results reveal a significant discrepancy, with 221 out of 644 repairs successfully passing functional tests yet failing to meet review constraints, highlighting the limitations of functional-only evaluations in real-world software development contexts.
Functional success in coding agents is misleading鈥攐ver 34% of patches that pass tests still fail to meet critical review constraints.
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents'ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.
