Search papers, labs, and topics across Lattice.
This study examines the impact of instruction files on the performance of AI-agents in generating pull requests (Agentic-PRs) by analyzing 15,549 PRs across 148 projects in the AIDev dataset. The findings reveal a mixed effect, with 27.7% of projects experiencing a significant increase in merge rates while 26.35% saw a decline, indicating that simply providing instructions does not guarantee improved outcomes. Notably, projects that succeeded in enhancing their merge rates tended to have longer and better-structured instruction files, suggesting that the quality of instructions plays a crucial role in agent performance.
Instruction files can either enhance or hinder AI-agent performance in code generation, challenging the assumption that more guidance always leads to better outcomes.
AI-agents (e.g., GitHub Copilot) collaborate as teammates in different software engineering tasks, including code generation proposed through pull requests (Agentic-PRs). For better agent efficiency, developers create instruction files that guide the AI-agents, including how to navigate the project, locate the right components, run tests, respect best practices, and more. In this paper, we investigate the relationship between the creation of these instructions and the performance of AI-agents in creating better pull requests, which have a higher chance of success (i.e., the merge rate), address more complex tasks (e.g., code churn), and require less effort to be merged (e.g., time to merge). To this end, we analyze 15,549 agentic PRs from 148 projects in the AIDev dataset. Using the three dimensions, we compare each project before and after the creation of the instruction files. We find that specifying instructions for AI-agents does not necessarily lead to better results. With the instruction files, 27.7\% of the projects increased their merge rate by at least 20\%, while 26.35\% decreased it. The same observation is seen with the amount of changes (e.g., code churn, number of modified files) and with the efforts to merge an agentic PR (e.g., merge time and number of comments). From a first exploration, we find that projects that managed to increase their merge rate have substantially longer instruction files, which are also well structured into a higher number of sections and sub-sections. Our results motivate the need for research to assist practitioners in framing the development of instruction files as a software engineering activity (aka, \textbf{Instructions-as-Code}).
