Search papers, labs, and topics across Lattice.
This paper examines the application of Generative AI (GenAI) to qualitative research within software engineering, highlighting both its potential and limitations across diverse research strategies. It reviews existing empirical evidence of GenAI assistance, weighs the advantages and disadvantages of GenAI-mediated approaches, and re-evaluates qualitative research quality factors in light of GenAI. The authors aim to provide researchers with a balanced perspective on the use of GenAI in this context, cautioning against overgeneralization and emphasizing the need for careful adaptation.
Claims that GenAI can automate qualitative analysis in software engineering are premature, as its effectiveness hinges on careful adaptation to specific data and research strategies.
Qualitative research gives rich insights into the quintessentially human aspects of software engineering as a socio-technical system. Qualitative research spans diverse strategies and methods, from interpretivist, in situ observational field studies, to deductive coding of data from mining studies. Advances in large language models and generative AI (GenAI) have prompted claims that artificial intelligence could automate qualitative analysis. Such claims are overgeneralizing from narrow successes. GenAI support must be carefully adapted to the data of interest, but also to the characteristics of a particular research strategy. In this Frontiers of SE paper, we discuss the emerging use of GenAI in relation to the broad spectrum of qualitative research in software engineering. We outline the dimensions of qualitative work in software engineering, review emerging empirical evidence for GenAI assistance, examine the pros and cons of GenAI-mediated qualitative research practices, and revisit qualitative research quality factors, in light of GenAI. Our goal is to inform researchers about the promises and pitfalls of GenAI-assisted qualitative research. We conclude with future plans to advance understanding of its use in software engineering.
