Search papers, labs, and topics across Lattice.
This study investigates the impact of varying definitions of media bias on annotation outcomes by conducting a between-subjects experiment with 354 participants and evaluating four large language models (LLMs) on six news articles across four bias categories. The findings reveal that the conceptual framing of definitions significantly influences annotation divergence among both humans and LLMs, while elaboration that preserves the construct does not have the same effect. This highlights the critical importance of clear and consistent definitional frameworks in bias detection, as it may affect the reliability of model training and evaluation in this domain.
Conceptual framing of bias definitions can lead to significant discrepancies in annotation, impacting both human and LLM assessments.
Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain implicit, even when given the same name. Such variation makes it unclear whether models trained for the same bias category learn the same construct or different phenomena, a problem largely overlooked in prior work. We examine how definition choice affects bias annotation in a between-subjects experiment with 354 participants and a parallel evaluation with four LLMs. Participants and models rate six news articles across four bias categories using definitions that vary in conceptual framing and elaboration. Across 8,496 human and 28,800 LLM ratings, we find that the conceptual target of a definition drives annotation divergence, while construct-preserving elaboration does not: conceptual framing significantly shifts annotations for humans and does so even more strongly for LLMs. We discuss implications for construct specification in annotation protocols and prompt-based measurement, and consider how definitional sensitivity may propagate to downstream classification beyond media bias. We also release MUDD, the Multi-Definition Bias Detection Dataset.
