Search papers, labs, and topics across Lattice.
This paper introduces SWE-NFI, a benchmark designed to evaluate coding agents on their ability to make non-functional improvements (NFIs) while preserving behavior, addressing a significant gap in existing evaluations that focus primarily on functional correctness. The benchmark consists of 188 tasks derived from real pull requests in open-source Python projects and operationalizes NFIs into 92 executable rules, allowing for a comprehensive assessment of coding agents. Results indicate that while the top-performing agent achieves a 70.0% functional correctness rate, all agents significantly underperform compared to human developers, especially in structural improvements, highlighting the need for enhanced capabilities in coding agents.
Coding agents struggle with non-functional improvements, scoring as low as 1.3 on structural changes compared to human developers' 1.5.
Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents'NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.
