Search papers, labs, and topics across Lattice.
This experience report evaluates the use of a customized MyGPT to conduct semi-structured interviews in empirical software engineering (ESE) studies, focusing on refactoring practices and generative AI in Scrum activities. The AI-driven approach allowed participants to engage in self-administered interviews in their preferred language, yielding a high compliance rate with the expected synthesis format and positive participant feedback. While the method demonstrated operational viability and acceptability, it highlighted limitations such as generic questioning and privacy concerns, suggesting that AI interviewers are best suited for low-risk, focused studies with careful oversight.
AI-driven interviews achieved a 90.9% positive experience rating, but raised concerns about depth and privacy that could redefine ESE methodologies.
Semi-structured interviews are widely used in empirical software engineering (ESE), but they are resource-intensive and difficult to coordinate across schedules, locations, and natural languages. This experience report examines a customized MyGPT used to conduct short, self-administered interviews in two ESE studies: one on refactoring practices and another on generative AI in Scrum-related activities. Participants accessed the interviewer through shared links, used voice interaction, selected a preferred natural language, and completed the interview without a researcher present. The AI followed a predefined protocol and generated a structured synthesis that participants voluntarily submitted; these artifacts were not treated as verbatim transcripts. We analyzed 66 submissions and questionnaire responses, and audited artifact format, language, length, and protocol consistency. Of the submitted artifacts, 92.4% followed the expected synthesis format, 65 were predominantly in Portuguese and one in English, and two conflicted with the reported protocol. Participants generally rated the experience positively: 90.9% reported a positive overall experience and comfort, 95.5% considered the questions clear, 97.0% rated the pace positively, and 89.4% would participate again. Reported limitations included generic questions, limited sensitivity to answers, insufficient depth, privacy concerns, and missed human interaction. The findings support the operational viability and acceptability of this workflow among analyzed respondents, but do not establish completion rates, time savings, summary fidelity, or equivalence to human-conducted interviews. AI interviewers should therefore be treated as a complementary option for short, focused, low-risk studies, with protocol design, privacy guidance, artifact validation, and human oversight.
