Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces VisER, a training-free metric that detects object hallucinations in large vision-language models (LVLMs) by evaluating generated object mentions from two perspectives: Visual Evidence and Visual Reliance. By assessing the compatibility of object-context relationships with specific visual evidence and the degree of support from the image versus the generated text, VisER provides a more nuanced grounding score that addresses the limitations of existing detection methods. The results show that VisER significantly enhances AUROC and AUPR metrics across various LVLMs and benchmarks, indicating improved reliability in object recognition tasks.
Object hallucination detection can be revolutionized by a two-sided metric that distinguishes between visual evidence and textual support, leading to more reliable LVLM outputs.
Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.