Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces TADDLE, a tool-augmented agent designed to detect deficiencies in LLM-generated peer reviews, addressing the challenge of uniformly fluent but potentially flawed reviews. TADDLE uses four specialized tools (Verify, Correct, Complete, Transform) orchestrated by an agent to identify specific defect types within reviews. Experiments on a new expert-annotated benchmark of 1,800 reviews demonstrate TADDLE's strong performance in both binary detection and multi-label classification of review defects.
LLM-generated peer reviews may look good, but TADDLE reveals their hidden flaws, outperforming existing methods in detecting specific defect types.
LLM-generated peer reviews are increasingly common at major venues, yet their deficiencies are hard to detect because they are uniformly fluent and well-structured. Existing work either classifies authorship without judging quality, or scores quality with features designed for human-written reviews; no prior system detects deficiencies in LLM-generated reviews at the level of individual defect types. To bridge the gap, we introduce TADDLE, a Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews, together with the first expert-annotated benchmark for this task. Our benchmark comprises 1,800 reviews on 50 ICLR 2025 papers, multi-label-annotated by 18 domain experts against a taxonomy of six defect categories (plus a non-deficient label). TADDLE decomposes detection into four specialized analysis tools -- Verify, Correct, Complete, and Transform -- orchestrated by an agent; an integrator synthesizes their outputs into binary and multi-label classifications via two-stage semi-supervised learning. Extensive experiments show that TADDLE performs strongly on both binary detection and the multi-label classification task. We release the benchmark and code at https://github.com/AquariusAQ/TADDLE.