Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces three novel protocols for aggregating confidence signals in multiagent systems, addressing the lack of a unified confidence measure for outputs. By transforming raw confidence signals for comparability and employing methods like soft voting and Bayesian fusion, the authors demonstrate that their aggregated confidence significantly outperforms individual agent confidence and standard debate baselines in terms of discriminative ability while maintaining stable correctness. The findings reveal that calibration enhances F1-scores across estimators, indicating a robust approach to improving decision-making in multiagent NLP tasks.
Aggregated confidence signals from multiagent systems can dramatically enhance decision reliability without sacrificing output correctness.
Confidence is used for reliability, oversight, and a range of downstream decision tasks in Natural Language Processing (NLP), yet no existing method produces or evaluates a confidence for the output of a multiagent system. Prior work uses confidence within multiagent debate (MAD) to weight messages, trigger debate, or calibrate individual agents, but it never aggregates these into a single confidence for the system itself. We introduce three protocols that produce a final answer along with a single aggregated confidence by first transforming raw confidence signals to make them comparable across models, then combining them via soft voting or a probability fusion we call Bayesian fusion. This aggregated confidence is substantially more discriminative (AUARC) than that of the best single agent or the standard debate baselines, while correctness (F1-score) stays stable and recovers the losses MAD incurs on more ambiguous tasks. Analyzing two estimators, sequence probability and self-report, alongside parametric and non-parametric calibrators, we find that calibration improves F1 for both estimators while AUARC is less reliant on it. We evaluate six homogeneous and heterogeneous debating pairs per benchmark, across five benchmarks and four task types, spanning a range of model capabilities and sizes.