Search papers, labs, and topics across Lattice.
This study investigates how rhetorical choices in scientific manuscripts influence AI-based peer review outcomes, revealing that certain rhetorical dimensions significantly affect AI reviewers' judgments. By analyzing a controlled corpus of 4,200 manuscripts, the authors find that evidence framing and novelty stance create the most pronounced contrasts in assessments, while other dimensions have less stable effects. The results indicate that the AI reviewer's initial score heavily influences score adjustments, with lower scores benefiting more from rhetorical enhancements than higher scores, highlighting the complex interplay between manuscript presentation and review outcomes.
Rhetorical framing can dramatically shift AI reviewers' scores, with evidence and novelty standing out as key influencers.
As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions. Two LLM rewriters transform six rhetorical dimensions in opposing directions, and five LLM reviewers evaluate the resulting manuscripts under standard and strict protocols. We also test joint, recursive, and reviewer-guided rewriting. Our results show that rhetorical sensitivity is structured rather than uniform. Evidence framing and novelty stance produce the largest positive-negative contrasts in overall assessment, with scope framing forming a weaker second tier; the remaining dimensions have smaller or less stable effects. This hierarchy persists across human-assessed quality levels, but score movement depends strongly on the AI reviewer's original score: lower scores tend to rise, higher scores tend to fall, and directional contrasts are clearest in the middle ranges. More elaborate workflows do not reliably yield larger gains. Joint rewriting is strongly rewriter-dependent, reviewer guidance does not consistently outperform an unguided second pass, and repeated rewriting yields diminishing, configuration-dependent returns. Across conditions, the rewriter primarily determines the separation between opposing variants, whereas the reviewer determines the magnitude and sign of their score effects. Strict review lowers mean OA by 1.36 points without consistently changing rhetorical sensitivity. These findings identify when rhetorical presentation influences AI scientific review and motivate evaluation systems robust to content-preserving variation in scientific writing.
