Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study quantifies the prevalence of large language model (LLM) usage among survey respondents on crowdsourcing platforms and evaluates strategies to detect and mitigate this behavior. By analyzing responses from 250 participants across different platforms, the researchers found that LLM-assisted responses ranged from under 10% on Prolific to over 80% on Mechanical Turk, with mitigation measures reducing LLM usage but not necessarily enhancing data quality. The study highlights the need for researchers to actively screen for LLM usage through keystroke analysis and tailored survey design to maintain data integrity.
LLM usage in survey responses can exceed 80% on some platforms, raising serious concerns about data validity.
Large language models are increasingly used by participants on crowdsourcing platforms when responding to surveys, potentially undermining the validity of collected data. Our study aims to quantify the prevalence of this behavior and investigate methods to detect and prevent it. In a series of surveys (N = 250), we examined conditions such as platform choice, survey length, requests not to use AI, and disabling copy-paste functionality. We were able to identify distinct characteristics of LLM-assisted responses and found that their frequency varied widely, from under 10% on Prolific to over 80% on Mechanical Turk. Mitigation measures reduced LLM usage but did not necessarily improve data quality. No participants employed browser-use agents at the time of our survey, but we report on our own detection experiments. We recommend that researchers actively screen survey responses for LLM usage by recording and analyzing keystroke data and crafting instructions and questions aimed at AI.