Search papers, labs, and topics across Lattice.
The paper investigates biases in AI-generated text detection within scientific publications by analyzing stylistic differences across countries and fields. It demonstrates that a single, pooled benchmark for AI-likeness confounds pre-existing stylistic variations with AI-generated text, leading to inaccurate estimations of AI use. By constructing and applying country-field-specific benchmarks, the study provides a more accurate and equitable assessment of AI's influence on scientific writing.
Naive AI detection methods can falsely flag human writing as AI-generated simply because of stylistic differences between countries and fields.
This paper examines how estimates of AI use in scientific writing can be biased when evaluation methods ignore contextual differences across countries and fields. Using large-scale data on journal publications from Dimensions, we construct AI-likeness benchmarks based on differences between human-written and LLM-rephrased abstracts. We show that a pooled benchmark may confound pre-existing stylistic variation with AI-generated text, producing substantial distortions across country-field groups even in pre-LLM publications. In contrast, country-field-specific benchmarks attenuate such distortions and provide a more credible baseline for comparison. Applying these methods to publications in 2025 reveals that the pooled benchmark systematically overestimates AI use in certain countries and fields while underestimating it in others. These findings highlight the importance of context-aware measurement for accurate and equitable evaluation of AI use in science.
