Search papers, labs, and topics across Lattice.
This paper investigates when LLMs can outperform human annotators in predicting aggregate subgroup opinions on subjective tasks. It finds that LLMs can be statistically superior estimators of group-level judgments due to their low variance and reduced bias coupling, even outperforming in-group human annotators under certain conditions. The analysis identifies regimes where LLMs are superior and highlights the limits where human judgment remains essential.
LLMs aren't just cheaper annotators; they can actually be *better* than humans at predicting aggregate opinions in certain subjective tasks.
Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives.
