Search papers, labs, and topics across Lattice.
The paper introduces VAUQ, a vision-aware uncertainty quantification framework for LVLM self-evaluation, addressing the limitations of language-prior-dependent methods in vision-conditioned prediction scenarios. VAUQ uses an Image-Information Score (IS) to quantify the reduction in predictive uncertainty due to visual input, coupled with a core-region masking strategy to emphasize salient regions. Experiments demonstrate that VAUQ's training-free scoring function, combining predictive entropy and core-masked IS, surpasses existing self-evaluation methods in reflecting answer correctness across various datasets.
LVLMs can now better judge their own vision-based answers, thanks to a new method that focuses on how much they actually "see" in the image.
Large Vision-Language Models (LVLMs) frequently hallucinate, limiting their safe deployment in real-world applications. Existing LLM self-evaluation methods rely on a model's ability to estimate the correctness of its own outputs, which can improve deployment reliability; however, they depend heavily on language priors and are therefore ill-suited for evaluating vision-conditioned predictions. We propose VAUQ, a vision-aware uncertainty quantification framework for LVLM self-evaluation that explicitly measures how strongly a model's output depends on visual evidence. VAUQ introduces the Image-Information Score (IS), which captures the reduction in predictive uncertainty attributable to visual input, and an unsupervised core-region masking strategy that amplifies the influence of salient regions. Combining predictive entropy with this core-masked IS yields a training-free scoring function that reliably reflects answer correctness. Comprehensive experiments show that VAUQ consistently outperforms existing self-evaluation methods across multiple datasets.
