Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the relationship between log-probability distributions, LLM-as-judge scores, and final task accuracy in multi-agent debate systems, which are often evaluated solely on their final answers. By analyzing a two-agent debate architecture consisting of a Constructor and an Auditor, the authors find that internal confidence signals, particularly from the Constructor, are strong predictors of reasoning quality and task correctness. The study reveals a significant asymmetry in confidence alignment, with the Constructor demonstrating a much stronger correlation with judged reasoning quality compared to the Auditor, highlighting critical reasoning failures more reliably.
Confidence signals in multi-agent debate systems can predict reasoning quality, but they reveal a surprising asymmetry favoring the Constructor over the Auditor.
Multi-agent debate systems are typically evaluated only on whether the final answer is correct, overlooking the quality of the intermediate reasoning that debate is designed to produce. This paper studies the relationship between three signals in multi-agent debate: token-level log-probability distributions over reasoning tokens, LLM-as-judge rubric scores assigned to those tokens, and final task accuracy. We examine whether internal confidence signals predict externally evaluated reasoning quality, and whether either signal aligns with task correctness, across three domains: rubric-based scoring, mathematical reasoning, and factual question answering. Our framework pairs a two-agent debate architecture -- a Constructor and an Auditor -- with an LLM-as-judge that scores each agent's reasoning along instruction following, justification quality, and evidence grounding, together with a critical-failure flag. Experiments in the rubric-scoring domain reveal a consistent four-phase confidence trajectory and a substantial role asymmetry: confidence aligns with judged reasoning quality roughly twice as strongly for the Constructor as for the Auditor, and confidence-based detection of critical reasoning failures is markedly more reliable for the Constructor (AUROC 0.804) than for the Auditor (0.634). These findings motivate the broader cross-domain investigation proposed in this paper.