Search papers, labs, and topics across Lattice.
This study investigates how software engineers interact with code labeled as generated by large language models (LLMs) during code reviews, utilizing a Wizard-of-Oz experimental design combined with eye-tracking technology. The findings reveal that while the thoroughness of reviews remained unchanged, participants exhibited increased fixation times on LLM-labeled code, suggesting that labeling influences attention and review strategies. This highlights a disconnect between developers' intentions and their actual review behaviors, prompting a reevaluation of AI policies in software development to enhance support for reviewing LLM-generated code.
Labeling code as LLM-generated significantly alters developers' attention and review strategies, revealing a critical gap between intention and behavior in code reviews.
Modern software development increasingly involves the use of large language models (LLMs) to generate code. Despite their rapid advancement, LLMs remain prone to errors and hallucinations, emphasizing the importance of careful code inspection. However, in practice, developers' trust in LLM-generated code and their willingness to review it thoroughly may differ from these recommendations. How developers actually behave when reviewing LLM-generated code remains largely unexplored. In this study, we conduct a Wizard-of-Oz experiment to examine how software engineers behave when code is explicitly labeled as LLM-generated during a code review task. We collect both behavioral data and participant feedback through eye-tracking and exit interviews. Combining Bayesian data analysis with qualitative analysis, we found that while the thoroughness of code review did not change for participants, they spent more time fixating on LLM-labelled code, indicating that the label itself influences attention. Practitioners also adapted their review strategy for LLM-labelled code by assessing the code based on specific criteria (e.g., logical correctness), or using the prompt to guide their review. These findings inform LLM-based tool design on labelling while incorporating the prompt as a software artifact. Our study reveals a gap between reviewers' intentions and actual reviewing behaviour, highlighting the need for software companies to revisit their AI policies (particularly regarding LLM-assisted development) to better support developers in reviewing LLM-generated code.
