Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces BiG-SURE, a novel uncertainty estimator that leverages a bipartite graph framework to assess semantic uncertainty in large language models (LLMs) and vision-language models (VLMs) without requiring access to model parameters. By utilizing cross-temperature semantic agreement, the method stabilizes low-temperature responses as anchors and probes high-temperature responses through meaning-preserving transformations, ultimately quantifying uncertainty via a Semantic Uncertainty and Reliability Estimation (SURE) score. Experimental results demonstrate that BiG-SURE significantly enhances average abstention AUROC across various text and multimodal QA tasks compared to existing black-box uncertainty estimators.
BiG-SURE achieves superior uncertainty estimation in black-box LLMs, outperforming traditional methods while remaining simple and unsupervised.
Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.