Search papers, labs, and topics across Lattice.
The paper introduces LR-bench, a new benchmark for reviewer assignment comprising 1055 expert-annotated paper-reviewer-score annotations from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings. It then proposes RATE, a reviewer-centric ranking framework that distills reviewer publications into keyword profiles and fine-tunes an embedding model using weak supervision from heuristic retrieval signals. Experiments on LR-bench and the CMU dataset demonstrate that RATE achieves state-of-the-art performance compared to strong embedding baselines.
Forget outdated benchmarks: LR-bench offers a fresh, 2024-2025-era dataset for reviewer assignment, and RATE leverages reviewer profiles for state-of-the-art matching.
Reviewer assignment is increasingly critical yet challenging in the LLM era, where rapid topic shifts render many pre-2023 benchmarks outdated and where proxy signals poorly reflect true reviewer familiarity. We address this evaluation bottleneck by introducing LR-bench, a high-fidelity, up-to-date benchmark curated from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings collected via a large-scale email survey, yielding 1055 expert-annotated paper-reviewer-score annotations. We further propose RATE, a reviewer-centric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles and fine-tunes an embedding model with weak preference supervision constructed from heuristic retrieval signals, enabling matching each manuscript against a reviewer profile directly. Across LR-bench and the CMU gold-standard dataset, our approach consistently achieves state-of-the-art performance, outperforming strong embedding baselines by a clear margin. We release LR-bench at https://huggingface.co/datasets/Gnociew/LR-bench, and a GitHub repository at https://github.com/Gnociew/RATE-Reviewer-Assign.
