Search papers, labs, and topics across Lattice.
To disentangle changing human evaluation norms from shifts in paper distributions, the authors evaluated 32,638 ICLR submissions (2018–2025) by pairing 124,615 historical human reviews with 81,850 counterfactual reviews generated in a single window by a frozen LLM rater. Between 2018 and 2025, human score premiums for non-domain lexical complexity collapsed from +0.142 to -0.015, whereas the frozen machine rater's valuation remained unchanged (+0.080 to +0.082), yielding a significant difference-in-differences shift of -0.0100 ($q=0.013$). These findings show that peer reviewers actively discount surface-level signals once LLMs collapse their production cost, leaving static LLM judges calibrated on historical human feedback systematically misaligned with current human taste.
Human reviewers completely stopped rewarding flowery academic prose once LLMs made it cheap to fake, but LLM evaluators trained on historical data still pay a premium for it—revealing a hidden vulnerability in static LLM-as-a-judge pipelines.
Large language models have collapsed the cost of producing lexically elaborate prose, and whether peer reviewers still reward it is a question about the evaluator, not about the text. When the association between a writing cue and review scores moves across years, the reviewers may have changed, the submissions may have changed, or both, and a regression of scores on text cannot say which. We separate the two with a frozen rater: 81,850 machine reviews of ICLR submissions from 2018 to 2025, all generated in one February-April 2025 window with one model family and one prompt, so that its year-to-year coefficients track submission composition alone and the human-minus-frozen trend difference identifies reviewer preference drift. On 32,638 submissions with 124,615 human reviews, the human coefficient on non-domain lexical complexity falls from +0.142 to -0.015 while the frozen rater moves from +0.080 to +0.082; the three-way difference-in-differences is -0.0100 (q=0.013), and forty random-wordlist placebos through the same specification centre on zero. Humans still reward sentence-length variability, which the frozen rater never registers, while the frozen rater still pays for lexical complexity at its earlier rate. Every claim is held to a double gate of false-discovery control and interval exclusion, and the findings that failed adversarial re-testing are reported. Reviewers discounted a cue whose production cost collapsed, as models of manipulable signals prescribe; an LLM judge calibrated to historical human preferences inherits the earlier schedule and drifts out of alignment while its agreement with humans on totals stays ordinary.
