Search papers, labs, and topics across Lattice.
This study systematically evaluates the reliability of LLM-as-a-Judge (LaaJ) for rubric verification in agentic scenarios, introducing RuVerBench, a benchmark designed for this purpose. The evaluation encompasses 2,458 instances across deep research and agentic coding domains, revealing that while advanced LLMs perform well, they still exhibit significant variability in scoring accuracy. Key findings highlight that weaker models are more affected by prompt variations, and strategies like batching and majority voting impact both accuracy and efficiency in rubric verification.
Even the most advanced LLMs struggle with consistent rubric verification, revealing substantial noise in scoring outputs across complex agentic scenarios.
Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric scoring remains underexplored. This concern is especially pronounced in agentic scenarios, where long, complex outputs further challenge reliable scoring. To address this, we conduct a systematic meta-evaluation of LaaJ reliability for rubric verification. We introduce RuVerBench, the first benchmark for assessing LaaJ reliability in rubric verification for agentic scenarios. RuVerBench covers two prevalent agentic domains, deep research and agentic coding, with 2,458 instances, each containing a model-generated output, a rubric, and a human-annotated label indicating whether the output satisfies the rubric. Using RuVerBench, we evaluate numerous frontier LLMs and find that even the most advanced models achieve strong performance but still exhibit substantial noise. We further analyze the impact of key LaaJ strategies, including prompt design, batching, and majority voting, on rubric verification. We find that weaker models are more sensitive to prompt variations, batched verification presents a trade-off between accuracy and efficiency, and majority voting yields effective but diminishing returns. We have released our dataset and code to facilitate future research: https://github.com/THU-KEG/RuVerBench.
Tsinghua AI