Search papers, labs, and topics across Lattice.
This paper compares the frequency of breaking changes introduced by AI agents versus human developers in pull requests (PRs) within Python repositories. Analyzing 7,191 agent-generated and 1,402 human-authored PRs using AST-based analysis, the study finds that agents introduce fewer breaking changes overall in code generation tasks. However, agents exhibit a significantly higher risk of introducing breaking changes during maintenance tasks like refactoring and chore changes, particularly when exhibiting high confidence.
AI coding agents are less likely to break your code *except* when they're confidently "maintaining" it, where they're actually twice as risky as humans.
AI coding agents are increasingly integrated into modern software engineering workflows, actively collaborating with human developers to create pull requests (PRs) in open-source repositories. Although coding agents improve developer productivity, they often generate code with more bugs and security issues than human-authored code. While human-authored PRs often break backward compatibility, leading to breaking changes, the potential for agentic PRs to introduce breaking changes remains underexplored. The goal of this paper is to help developers and researchers evaluate the reliability of AI-generated PRs by examining the frequency and task contexts in which AI agents introduce breaking changes. We conduct a comparative analysis of 7,191 agent-generated PRs with 1402 human-authored PRs from Python repositories in the AIDev dataset. We develop a tool that analyzes code changes in commits corresponding to the agentic PRs and leverages an abstract syntax tree (AST) based analysis to detect potential breaking changes. Our findings show that AI agents introduce fewer breaking changes overall than humans (3.45% vs. 7.40%) in code generation tasks. However, agents exhibit substantially higher risk during maintenance tasks, with refactoring and chore changes introducing breaking changes at rates of 6.72% and 9.35%, respectively. We also identify a "Confidence Trap" where highly confident agentic PRs still introduce breaking changes, indicating the need for stricter review during maintenance oriented changes regardless of reported confidence score.
