Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of supervised uncertainty quantification (UQ) ensembles in detecting hallucinations in large language models (LLMs) across various generation contexts. By training a classifier on heterogeneous UQ outputs from a small, domain-specific dataset, the authors demonstrate that these ensembles significantly enhance hallucination detection, outperforming individual scorers in 30 out of 32 scenarios tested. Notably, the ensembles maintain their performance advantage even when faced with distribution shifts, indicating robust applicability across different datasets and generation regimes.
Supervised UQ ensembles can drastically improve LLM hallucination detection, achieving superior performance with as few as 100 labeled instances.
Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit.
