Search papers, labs, and topics across Lattice.
This study examines the impact of fraudulent or AI-assisted responses on the validity of software engineering surveys by analyzing four datasets through both manual and automated detection methods. While quantitative results remained largely stable after filtering out suspicious responses, qualitative findings revealed significant shifts based on contextual framing and evidence interpretation. The research underscores the necessity of employing multiple validation techniques, especially for studies that depend on open-ended responses, to ensure accurate insights into participant experiences.
AI-generated responses can subtly distort qualitative survey findings, challenging the reliability of participant insights in software engineering research.
Background: Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI assisted participation in software engineering surveys. Aims: This study investigates how suspicious or potentially AI assisted responses may affect the validity of software engineering survey findings. Method: We conducted a secondary analysis of four software engineering survey datasets using manual identification of suspicious responses, automated AI generated text detection, descriptive statistical analysis, and thematic analysis. We compared findings obtained from the original and manually cleaned datasets. Results: Quantitative findings generally remained stable after filtering suspicious responses, although some demographic and analytical variables showed moderate variation, affecting the interpretation of specific participant groups and contextual characteristics. In contrast, qualitative findings were more strongly influenced by changes in contextual framing, code prominence, and the nature of the evidence supporting interpretation, shaping how participants'experiences and study contexts were interpreted and characterized. Conclusions: AI assisted participation may influence software engineering survey findings differently depending on the type of analysis being conducted. The findings reinforce the importance of combining multiple validation procedures, particularly in studies relying on open ended responses.
