Search papers, labs, and topics across Lattice.
This paper introduces Dude, a Dual-Detection Multi-Agent System designed to enhance the detection of discrepancies between research papers and their corresponding code, addressing the limitations of existing single-agent LLM approaches. By implementing a granularity-aligned negotiation process and a two-stage salience-filtering mechanism, Dude significantly reduces false positives while improving recall and precision in discrepancy detection. Experimental results demonstrate that Dude achieves up to a 22.8% increase in recall and an 18.7% improvement in F1 score compared to baseline methods, marking a substantial advancement in automated paper-code review processes.
Dude achieves up to 22.8% better recall in paper-code discrepancy detection, tackling the critical challenge of false positives in multi-agent systems.
LLM-empowered paper-code discrepancy detection has received growing concern since the scaling of research submissions exceeds the manual review capability. However, the limited context capacity and one-sided discrepancy detection of existing single-agent LLM paradigms lead to an inferior recall performance in detecting discrepancies. In this paper, we propose Dude, the first Dual-Detection Multi-Agent System for paper-code discrepancy detection. We discover that the granularity asymmetry of the paper-language and code-language introduces over-interpretation and over-reporting challenges in a multi-agent system design for discrepancy detection, resulting in increasing false positives. To address this, we propose a granularity-aligned negotiation and a two-stage salience-filtering mechanism in Dude, which effectively prevents agents from falsely reporting discrepancies. Experimental results in real-world paper-code discrepancy datasets showcase Dude's significant recall and precision improvement by up to 22.8%, increasing F1 score by up to 18.7% compared to baseline methods.
