Search papers, labs, and topics across Lattice.
This paper introduces Credal Large Language Models (CLLMs), which utilize an ensemble of LoRA adapters to represent uncertainty through a credal set, allowing for a range of plausible predictive distributions rather than a single output. The authors derive two novel commitment scores鈥擟redal Token Commitment (CTC) and Semantic Commitment Consistency (SCC)鈥攖o enhance hallucination detection and improve QA accuracy while maintaining calibration. Evaluations show that CLLMs outperform standard models in various benchmarks, achieving high accuracy and low expected calibration error, particularly excelling in selective prediction tasks.
CLLMs achieve 99.0% accuracy on OpenBookQA while maintaining low calibration error, redefining how we handle uncertainty in LLMs.
Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation we derive two complementary commitment scores. Credal Token Commitment (CTC) is a token-space score that combines lower-bound support, credal width, and intersection entropy, computed without additional generation. Semantic Commitment Consistency (SCC) extends commitment to semantic space using sampled completions, with SCC-Gap measuring the mismatch between token-level and semantic-level support. We evaluate hallucination detection, calibration, selective prediction, and reasoning on Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B across OpenBookQA, CoQA, TriviaQA, and ARC-Challenge. CLLM is the best method on QA accuracy at competitive expected calibration error, and CTC tracks the best hallucination AUROC within 1.5 pp on most settings without additional generation. On selective prediction at 80% coverage, CLLM with SCC reaches 99.0% accuracy on OpenBookQA, and on ARC-Challenge CLLM with Csem confidence achieves <= 0.6% ECE across the three backbones.
