Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the predictive power of early-token confidence signals, specifically token-level log-probabilities, for assessing reasoning quality in multi-agent LLM debates. By employing a debate-based essay scoring framework and comparing these confidence proxies against traditional rubric-based judge scores, the authors find that early-token confidence is a superior predictor of reasoning quality, especially in the initial phases of generation. Notably, the research reveals that supportive reasoning aligns more closely with confidence metrics than adversarial critiques, highlighting the nuanced dynamics of agent roles in LLM interactions.
Early-token confidence can predict reasoning quality in LLM debates better than full-sequence statistics, revealing critical insights into the generation process.
Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.