Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Investigating whether LLM evaluators confound argument quality with speaker identity, the authors analyzed 2,976 evaluations across fixed policy texts paired with varying source attributions. This dynamic is critical for LLM-as-a-judge reliability, as automated arbiters must separate logical validity from the perceived credibility or ideological stereotype of the author. Across multiple frontier architectures—including reasoning-enabled configurations—evaluators exhibited significant source-position interaction effects, systematically downgrading identical arguments when attributed to counter-stereotypical sources and explicitly citing author-position mismatches in their verbal rationales.
LLM judges downgrade identical arguments simply because the speaker steps outside their expected political lane, directly penalizing counter-stereotypical sources even with reasoning modes enabled.
An argument can be surprising coming from a particular speaker without being a bad argument. Do AI evaluators keep these judgments apart? Two preregistered descriptive studies and a later Jev supplement collected 2,976 usable evaluations of six fixed texts about US AI policy, Germany's debt brake and Swiss nuclear energy. Each text was presented under several source attributions. The key comparison asks whether the gap between two sources changes when the argument changes. On Sol, for example, a national-security argument received mean ratings of 0.359 under CODEPINK and 0.639 under College Republicans; a civil-rights argument received 0.742 and 0.721. A constant preference for one source cannot explain that pattern. Related interactions appeared across topics and recent model configurations, including those with reasoning enabled, while several comparisons yielded small effects. The later European Jev supplement yielded five interactions below the adopted absolute reference of 0.05; its distinct rubric and interrupted collection limit comparison with the chat systems. Some written evaluations explicitly invoked a mismatch between a source and its attributed position. Taken together, the numerical and verbal evidence supports source-position coherence as a plausible explanation, alongside competing accounts involving credibility, authenticity and interpretation of the task. The paper develops this inference through controlled comparisons, reports conditional post hoc p-values in an appendix, and documents the human decisions and delegated checks behind an AI-conducted study.