Search papers, labs, and topics across Lattice.
This paper introduces Visual Semantic Entropy (VSE), a novel method for assessing visual ambiguity in vision-language models by perturbing only the image while keeping the text query fixed. The authors highlight that traditional entropy-based methods, such as Semantic Entropy, often underestimate uncertainty due to overconfident visual embeddings that suppress output diversity. Through extensive evaluation across multiple vision-language models and benchmarks, VSE is shown to outperform existing methods in capturing visual ambiguity and providing more accurate uncertainty estimates.
Overconfident visual embeddings can mask true uncertainty, but Visual Semantic Entropy reveals the hidden complexity of visual ambiguity in vision-language models.
Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions. Common entropy-based methods, such as Semantic Entropy (SE), rely on output diversity. Yet our analysis shows that overconfident visual embeddings suppress output diversity under stochastic decoding, causing SE to underestimate uncertainty in such cases. Recent methods instead probe output diversity through input perturbations, including textual paraphrasing or joint text-image perturbations, and show improved performance. We study these approaches and reveals that the resulting variability is often dominated by textual changes rather than visual evidence, causing uncertainty estimates to reflect prompt sensitivity rather than visual ambiguity. We therefore propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed. VSE measures uncertainty by clustering generated answers into semantic prototypes and computing the mass-weighted dispersion among them. Extensive evaluation across five modern vision-language models and five diverse VQA benchmarks demonstrates that VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.
