Search papers, labs, and topics across Lattice.
This paper introduces a taxonomy for fine-grained uncertainty quantification (UQ) in long-form language model outputs, categorizing methods based on response decomposition, unit-level scoring, and response-level aggregation. The authors formalize and extend several families of consistency-based black-box scorers within this framework. Experiments across multiple LLMs and datasets demonstrate that claim-response entailment outperforms more complex claim-level scorers, claim-level scoring is superior to sentence-level scoring, and uncertainty-aware decoding significantly improves factuality.
Long-form LLM factuality gets a boost: claim-response entailment and uncertainty-aware decoding are surprisingly effective at detecting hallucinations, outperforming more complex methods.
Uncertainty quantification has emerged as an effective approach to closed-book hallucination detection for LLMs, but existing methods are largely designed for short-form outputs and do not generalize well to long-form generation. We introduce a taxonomy for fine-grained uncertainty quantification in long-form LLM outputs that distinguishes methods by design choices at three stages: response decomposition, unit-level scoring, and response-level aggregation. We formalize several families of consistency-based black-box scorers, providing generalizations and extensions of existing methods. In our experiments across multiple LLMs and datasets, we find 1) claim-response entailment consistently performs better or on par with more complex claim-level scorers, 2) claim-level scoring generally yields better results than sentence-level scoring, and 3) uncertainty-aware decoding is highly effective for improving the factuality of long-form outputs. Our framework clarifies relationships between prior methods, enables apples-to-apples comparisons, and provides practical guidance for selecting components for fine-grained UQ.
