Search papers, labs, and topics across Lattice.
This study investigates the impact of prompt structure on pull request outcomes in LLM-assisted software development, analyzing 265 developer interactions with ChatGPT. By operationalizing prompt structure into three dimensions鈥擟ontext, Specificity, and Verification鈥攖he authors reveal that Specificity and Context significantly enhance actionable code generation, while Verification is crucial for code adoption. The findings highlight that prompt characteristics have distinct effects at different stages of the software engineering workflow, emphasizing the importance of tailored prompt design for effective AI integration.
Prompt structure can make or break the success of LLM-assisted code generation, with specific dimensions driving distinct outcomes in software development workflows.
Large language model (LLM)-powered tools such as ChatGPT are increasingly used in collaborative software engineering workflows, yet little is known about how prompt structure influences downstream pull request (PR) outcomes. Prior studies primarily examine conversational helpfulness, productivity, or coarse-grained adoption metrics, leaving the role of prompt structure in collaborative integration behavior insufficiently understood. We analyze 265 manually validated developer-ChatGPT interactions derived from self-admitted ChatGPT usage in open-source pull requests. Building on prior research on developer-facing artifacts and prompt engineering, we operationalize prompt structure using three dimensions: Context, Specificity, and Verification. We first evaluate whether LLM-assisted annotation can reliably reproduce human judgments of prompt structure, finding substantial variation across dimensions and workflow contexts. Specificity shows the most stable agreement with human judgments; Context is systematically under-scored by the LLM; and Verification remains difficult to assess consistently, motivating a hybrid human-LLM annotation strategy. Using this validated framework, we then examine how prompt structure influences actionable code generation, code adoption, and integration depth across AI-assisted PR workflows. Specificity and Context are most strongly associated with actionable code generation; Verification emerges as the primary predictor of code adoption; and integration depth is most strongly associated with Context. Overall, our findings show that prompt characteristics exert distinct, stage-dependent effects across AI-assisted software engineering workflows, influencing downstream adoption and integration through contextual grounding, task specificity, and evaluability cues.
