Search papers, labs, and topics across Lattice.
This paper systematically analyzes the validation practices of large language models (LLMs) in social science research by examining a comprehensive corpus of papers from eight leading journals. The study reveals that while LLM-generated measurements are increasingly central to empirical analyses, the validation practices employed are often inconsistent and inadequate, raising concerns about methodological rigor. By proposing complementary strategies for robust validation, the authors aim to establish better norms and standards for the use of LLMs in social science, addressing critical epistemic threats.
Inconsistent validation practices for LLM-generated measurements could undermine the integrity of social science research, highlighting an urgent need for improved standards.
Large language models (LLMs) are reshaping social science methodology. Researchers increasingly prompt language models to generate quantitative measurements of social concepts, for example labeling data or simulating survey responses. Yet LLMs pose methodological challenges including bias, hallucination, and brittleness across contexts, with unclear threats to validity. Standard practices and norms for addressing these challenges are still emerging. We collect and systematically analyze validation practices in a comprehensive corpus of papers from eight flagship social science journals that use LLMs as measurement instruments. We find that LLM-generated measurements frequently play a central role in empirical analyses, yet validation practices are inconsistent and limited. We outline complementary strategies for more robust validation, pointing toward better norms and standards around the use of LLMs in social science.
