Search papers, labs, and topics across Lattice.
This paper introduces Span-Level Uncertainty Estimation (SLUE), a novel approach that quantifies uncertainty in large language model (LLM) generation at the span level, addressing the limitations of existing token- and sequence-level methods. The authors present SPANUQ, a lightweight probe that efficiently distills uncertainty knowledge from multi-sample inference using a DETR-style span decoder, achieving significant improvements in uncertainty estimation quality and speed. Experimental results demonstrate that SPANUQ outperforms traditional methods, achieving an F1 score of 0.910 for span detection and enabling precise error localization that is unattainable with sequence-level approaches.
SpanUQ reveals that span-level uncertainty quantification can enhance both the accuracy and efficiency of LLM outputs, outperforming existing methods by a factor of 10-20x.
Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refinement in LLM generation. However, existing approaches operate at suboptimal granularities: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors. We formalize Span-Level Uncertainty Estimation (SLUE), a new task that targets the natural granularity for uncertainty: semantically coherent text spans, each conveying a single assessable unit of meaning. To address this task, we introduce SPANUQ, a lightweight probe that distills the uncertainty knowledge from expensive multi-sample inference into a single forward pass over LLM hidden states. SPANUQ employs a DETR-style span decoder to simultaneously detect spans and estimate their uncertainty via a Mixture of Beta distribution, trained with a principled combination of Beta NLL regression and contrastive ranking objectives. We construct SPANUQ-BENCH, the first span-level uncertainty benchmark comprising 20K prompts, 293K annotated spans, and continuous soft labels derived from multi-sample claim verification. Experiments on five LLM backbones show that SPANUQ consistently achieves the best span-level uncertainty quality, outperforming the strongest probe baseline and all sampling-based methods while being 10-20x faster. Its DETR-based span detector attains 0.910 F1, surpassing the best heuristic by 39.4%, enabling precise error localization that sequence-level methods cannot provide. The framework generalizes across five LLMs spanning two model families.
