Search papers, labs, and topics across Lattice.
This study investigates how large language models (LLMs) replicate the rhetorical intent of scientific citations compared to human authors. By employing a masked-citation task across 1,746 NLP conference papers, the authors find that LLMs cite less critically, favor older and more popular papers, and tend to reference authors who are socially distant from the citing scholar. These findings highlight a significant divergence in citation practices, suggesting that LLMs may inadvertently reinforce visibility bias while broadening the scope of cited work.
LLMs are reshaping scientific citation practices by citing less critically and favoring older, more popular works, potentially amplifying visibility bias in research.
Scientific citations carry rhetorical intent. Scholars may cite prior work positively (supporting), negatively (contrasting), or neutrally (mentioning). As large language models (LLMs) increasingly assist scientific writing, whether they reproduce citations with the same rhetorical intent as humans remains unclear. We introduce a masked-citation task to compare human and LLM-generated citation behavior. For each citation context, an LLM generates a replacement citation sentence, producing a counterfactual corpus directly comparable to human citation. We analyze what, whom, and how models cite, using an LLM-as-a-judge to classify citation intent and a 20-million-edge coauthorship network to measure social distance between cited authors. Across six popular LLMs and 1,746 top NLP conference papers (63k+ contexts, 132k+ citations), three patterns emerge: (1) Compared with human citation, LLMs cite significantly less critically; (2) LLMs over-cite popular and older papers, a tendency amplified for contrasting citations where human writing more often draws on recent, niche work; (3) Whereas humans often cite within their close social network, especially for supporting citations, LLMs tend to draw on more socially distant authors. Together, these differences are double-edged: LLM citation reaches beyond a scholar's close collaborators while being less critical and amplifying visibility bias, reshaping the rhetoric and reach of scientific citation.
