Search papers, labs, and topics across Lattice.
This paper analyzes 169 GitHub commits to understand how developers adopt LLM-generated code refactoring suggestions from ChatGPT. The study reveals that developers largely accept suggestions without modification, but when changes occur, they are substantial and follow five distinct patterns. These patterns are influenced by the refactoring task, the developer's prompt, and the quality of ChatGPT's response.
Developers overwhelmingly trust and directly apply LLM-generated code refactoring suggestions, but when they don't, the changes are surprisingly drastic and predictable.
Large language models (LLMs) have gained widespread popularity and have steadily improved over time, enabling software developers to use them for various code-related tasks. One common task is code refactoring, where the LLM suggests changes for the developer to apply to their code to improve quality attributes such as readability or maintainability. While current research focuses on evaluating LLM-generated refactoring suggestions, there is a limited understanding of how developers apply these suggestions in practice. To explore this, we analyze 169 GitHub commits where developers refactor their code based on a ChatGPT conversation linked in the commit message. We found that developers mostly accept and use the suggestions without modifications. When changes are made, they are mostly major and fall into five different patterns that depend on the refactoring activity, the developer's prompt, and the validity of the response from ChatGPT.
