Search papers, labs, and topics across Lattice.
This paper introduces CIC, a confidence-interval-based calibration framework that transforms arbitrary uncertainty scores from large language models (LLMs) into statistically grounded selective answering rules for question answering systems. By evaluating responses against an application-specific alignment criterion and estimating acceptance-conditioned error rates, CIC ensures that the selected thresholds maintain a controlled error rate while maximizing the answering rate. Experimental results across multiple LLMs demonstrate that CIC effectively balances risk management and answering efficiency, making it a robust solution for reliability-sensitive applications.
CIC guarantees controlled error rates in LLM responses while maximizing answering efficiency, a breakthrough for reliability-sensitive QA systems.
Large language models (LLMs) are increasingly deployed in question answering (QA) systems, yet they may generate hallucinated or misaligned responses without reliable confidence estimates. Uncertainty quantification (UQ) offers a natural basis for selective answering, where a system answers only when its prediction is deemed reliable and abstains otherwise. However, existing uncertainty scores for LLMs are often heuristic: a threshold chosen on such scores does not, by itself, provide statistical guarantees on the error rate among accepted answers. We propose CIC, a confidence-interval-based calibration framework that converts arbitrary uncertainty scores into risk-controlled selective answering rules. Given a held-out calibration set, CIC evaluates each generated response using an application-specific alignment criterion and associates it with an uncertainty score and a binary error label. For each candidate uncertainty threshold, CIC estimates the acceptance-conditioned error rate and constructs a high-probability upper confidence bound using either Hoeffding-style or Clopper-Pearson confidence intervals. It then selects the largest threshold whose upper bound is below a user-specified risk level $伪$, thereby maximizing the answering rate subject to a finite-sample reliability constraint. Under exchangeability, CIC guarantees with probability at least $1-未$ that the selected threshold, if non-null, controls the error rate among accepted answers at level $伪$. We evaluate CIC on both closed-ended and open-ended QA benchmarks across seven LLMs and multiple uncertainty estimators. Experimental results show that CIC consistently achieves valid risk control while retaining strong answering efficiency, providing a practical and statistically grounded mechanism for deploying LLMs in reliability-sensitive QA workflows.
