Search papers, labs, and topics across Lattice.
This paper introduces A-CRC-QA, a novel post-hoc calibration framework designed to enhance uncertainty-aware selective question answering in large language models. By reformulating selection-conditioned error control as a linear expectation constraint and employing a monotonized empirical-risk calibration inspired by conformal risk control, the method achieves asymptotic risk calibration without requiring additional training. Experimental results on CoQA and MedMCQA show that A-CRC-QA significantly improves the reliability of accepted answers while maintaining a high retention rate compared to existing uncalibrated and confidence-bound-based approaches.
A-CRC-QA achieves superior reliability in selective question answering by effectively controlling error rates without the need for retraining.
Large language models (LLMs) may generate fluent but incorrect answers, making uncertainty quantification important for reliable question answering. However, heuristic uncertainty scores cannot perfectly distinguish correct predictions from incorrect ones, and directly applying a fixed uncertainty threshold provides no statistical control over the error rate among accepted answers. To address this limitation, we propose A-CRC-QA, a post-hoc calibration framework for uncertainty-aware selective question answering. The proposed method reformulates selection-conditioned error control as a linear expectation constraint and applies a monotonized empirical-risk calibration procedure inspired by conformal risk control. Since the resulting instance-wise loss is generally non-monotone with respect to the acceptance threshold, our framework targets asymptotic rather than finite-sample risk control. A-CRC-QA is model-agnostic, requires no additional training, and can be combined with different uncertainty estimators. Experiments on CoQA and MedMCQA demonstrate its applicability to both open-ended and closed-ended question answering, achieving a favorable trade-off between accepted-answer reliability and answer retention compared with uncalibrated and confidence-bound-based baselines.
