Search papers, labs, and topics across Lattice.
This paper introduces matched trajectory replay, a novel protocol for evaluating the impact of confidence signals on decision-making in interactive language-model agents. By comparing raw confidence with isotonic calibration in a multi-hop question-answering context, the authors reveal that calibration can significantly enhance the accuracy of committed answers by up to 41 percentage points, albeit at the cost of reduced coverage and increased retrieval reliance. Ultimately, while calibration improves the interpretability of commitment risk, it does not effectively predict the utility of additional retrievals, highlighting the need for a separate value-of-information assessment in retrieval strategies.
Calibration can boost answer accuracy by up to 41 percentage points but may also lead to reduced coverage and increased reliance on retrieval.
Interactive language-model agents use confidence signals to decide whether to answer immediately, retrieve additional evidence (from memory or external knowledge), or defer. Yet confidence is usually evaluated in isolation, without measuring the trajectory-level consequences of the actions it triggers. We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings. The protocol holds candidate answer states, evidence points, budgets, and action costs fixed. We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system using Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. At the same numerical commitment threshold, calibration changes which questions agents ultimately commit to answering. Across all six model-dataset pairs, it increases accuracy among committed answers by up to 41 percentage points. However, it can reduce coverage and increase retrieval use. Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue. These effects reflect a shift to a more selective, lower-risk operating point, not improved answers or confidence ranking. A calibration map fitted before retrieval improves held-out calibration through retrieval depths one and two, but is worse than raw confidence at depth three for all three models. Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval. Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval. Retrieval therefore requires a separate value-of-information or utility estimate. Evaluations should report held-out calibration, risk-coverage, and retrieval cost.
