Search papers, labs, and topics across Lattice.
The paper introduces DREAM, a multi-round debate framework using LLM agents with opposing stances and iterative critique, to address the problem of incomplete relevance labels in IR benchmarks. DREAM achieves 95.2% labeling accuracy with only 3.5% human involvement by using agreement-based debate for accurate labeling and reliable AI-to-human escalation for uncertain cases. Using DREAM, the authors construct BRIDGE, a refined benchmark with 29,824 newly identified relevant chunks, demonstrating that incomplete labels distort retriever rankings and retrieval-generation alignment.
LLM debate resolves annotation gaps in IR benchmarks, uncovering nearly 30,000 missing relevant chunks and flipping the leaderboard for retrieval systems.
Information retrieval (IR) evaluation remains challenging due to incomplete IR benchmark datasets that contain unlabeled relevant chunks. While LLMs and LLM-human hybrid strategies reduce costly human effort, they remain prone to LLM overconfidence and ineffective AI-to-human escalation. To address this, we propose DREAM, a multi-round debate-based relevance assessment framework with LLM agents, built on opposing initial stances and iterative reciprocal critique. Through our agreement-based debate, it yields more accurate labeling for certain cases and more reliable AI-to-human escalation for uncertain ones, achieving 95.2% labeling accuracy with only 3.5% human involvement. Using DREAM, we build BRIDGE, a refined benchmark that mitigates evaluation bias and enables fairer retriever comparison by uncovering 29,824 missing relevant chunks. We then re-benchmark IR systems and extend evaluation to RAG, showing that unaddressed holes not only distort retriever rankings but also drive retrieval-generation misalignment. The relevance assessment framework is available at https: //github.com/DISL-Lab/DREAM-ICLR-26; and the BRIDGE dataset is available at https://github.com/DISL-Lab/BRIDGE-Benchmark.
