Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Standard LLM confidence estimators aggregate sequence likelihoods into global metrics, obscuring localized factual errors where an otherwise fluent generation hinges on a single hallucinated token. To resolve this dilution, TRACE models decoding-time uncertainty as a trajectory by capturing token-level surprisal and predictive entropy, then applying local operators to isolate transient risk spikes. Evaluated across seven LLMs and four tasks, the resulting estimator outperforms 19 baselines, improving AUROC from 0.764 to 0.817 and reducing Brier score from 0.136 to 0.120 without requiring multiple generation samples or external verifiers.
Global sequence probabilities routinely mask fatal single-token hallucinations, but capturing localized uncertainty spikes along the decoding trajectory boosts calibration AUROC to 0.817 in a single pass without extra sampling.
Reliable confidence estimation is essential for large language model deployment. However, answer-level calibration remains challenging because generation errors are often localized: a response may be fluent and high-probability overall while still failing at a critical number, entity, or factual claim. Existing estimators compress token probabilities, sequence likelihoods, entropy, or beam statistics into a global score, which can dilute such local risk signals. We propose TRACE, a single-pass, decoded-answer-preserving confidence estimator that treats decoding-time uncertainty as a trajectory through three steps: (i) recording token-level surprisal and predictive entropy during decoding, (ii) applying local risk operators to preserve uncertainty spikes, and (iii) converting localized trace risk into answer-level confidence. TRACE produces a label-free risk score, while TRACE+ calibrates trace-only features into probabilities using a held-out split, without extra generations or external verifiers. We evaluate four tasks against 19 calibration baselines, and TRACE+ reduces Brier from 0.149 to 0.137 and improves AUROC from 0.758 to 0.792 over the strongest likelihood baseline. Across seven LLMs, TRACE+ improves over the best non-TRACE baseline pool from 0.136 to 0.120 Brier and from 0.764 to 0.817 AUROC. Results show that localizing decoding-time risk provides a general approach to calibration.