Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study empirically evaluates the alignment of LLM-generated reviews with human reviews in the context of the 2025 ACL Rolling Review, revealing that while there is some reasonable alignment, it varies significantly across different prompts and models. The authors also explore how authors can leverage LLM reviews in an iterative draft-revise workflow, finding that this "gaming" approach can lead to statistically significant score improvements for up to 35% of papers. These findings highlight the complexities of LLM integration in academic review processes and the potential for authors to enhance their submissions through strategic use of LLM feedback.
LLM-generated reviews can boost paper scores by up to 35% when authors strategically revise based on feedback, but alignment with human reviews is inconsistent.
LLM-generated reviews for scientific papers are gaining considerable traction and are even being officially piloted by major conferences. We have to assume that not only reviewers are using LLM-assistance, but also that authors use LLMs to revise their papers before submitting. In this work, we perform empirical experiments on papers from the 2025 ACL Rolling Review (ARR) to evaluate LLM reviews from both the author and the reviewer perspective. First, we identify a limited alignment of LLM reviews with human ones. In the best-case scenario, the alignment is reasonable. However, we also find that LLM-human alignment varies substantially across prompts and models. Finally, we investigate the scenario in which the author uses an iterative draft-revise workflow to improve the submission according to the LLM review. We find that this"gaming"of LLM reviews can be effective in specific scenarios, leading to a statistically significant increase of overall scores for up to 35\% of papers. We publish our code: https://github.com/uhh-hcds/reviewarcade.