Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DEPBENCH, a benchmark designed to evaluate coding agents on 203 real-world dependency-upgrade tasks that involve hidden code-level changes requiring source code adaptation. The study reveals that mainstream coding agents only successfully complete 51.2% of these tasks, indicating a significant gap in their ability to handle the complexities of software maintenance in diverse ecosystems. These findings underscore the critical need for improved coding agent capabilities to ensure software reliability amidst dependency upgrades.
Mainstream coding agents only solve half of real-world dependency upgrade tasks, revealing a major gap in their effectiveness for software maintenance.
Modern software systems rely heavily on third-party dependencies, but upgrading those dependencies remains a costly maintenance activity. Dependency upgrades do not always preserve the function signatures, type systems, APIs, or runtime semantics assumed by existing code. Consequently, developers often need to perform source code adaptations to accommodate dependency-induced changes. However, such code-level changes are often not explicitly communicated to project maintainers, posing a significant challenge to software reliability. Meanwhile, coding agents have emerged as a new form of software development tool and are increasingly adopted by developers due to their automation capabilities. In this paper, we introduce DEPBENCH, a benchmark consisting of 203 real-world dependency-upgrade tasks across five package ecosystems spanning five language communities, each involving hidden code-level changes that require source code adaptation. We evaluate mainstream coding agents on DEPBENCH. The best completed configuration solves only 104/203 tasks (51.2%), with substantial variation across agent harnesses, models, and ecosystems, highlighting an important gap between current agent capabilities and real-world software maintenance needs.