Search papers, labs, and topics across Lattice.
This paper introduces Verifier-Guided Decoding (VGD), a novel framework that leverages an Intrinsic Grounding Signature (IGS) to selectively mitigate object hallucinations in large vision-language models (LVLMs). By employing a lightweight verifier to assess the reliability of object mentions during generation, VGD effectively suppresses high-risk hallucinations while maintaining the model's grounded object coverage and visual understanding. Experimental results demonstrate that VGD significantly reduces hallucinations by up to 43.6% without compromising the quality or length of generated captions.
Hallucinations in LVLMs can be cut by over 43% without sacrificing grounded object coverage, thanks to a novel verifier-guided approach.
Large vision-language models (LVLMs) often hallucinate objects that are absent from an image. Despite recent progress, existing mitigation methods still lack reliable object-level grounding diagnostics and therefore tend to apply coarse-grained interventions, which can impair visual understanding, shorten responses, and reduce coverage of genuinely grounded objects. The key challenge is thus to detect, during generation, whether each emerging object mention is supported by reliable visual evidence, so that hallucination can be mitigated selectively. Yet output confidence reflects next-token plausibility rather than visual support, allowing language priors to make absent objects appear certain. We show that the missing diagnostic evidence is encoded in an Intrinsic Grounding Signature (IGS), a distributed signed attention pattern that remains informative for such confident hallucinations. Based on IGS, we propose Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation. Because VGD intervenes only on object mentions identified as high risk, it reduces object hallucination while preserving the model's original visual understanding and grounded object coverage. Experiments on CHAIR and AMBER-G show that VGD achieves state-of-the-art object hallucination reduction: at @rec90, it cuts AMBER-G CHAIR by 43.6\% while retaining 99.6\% of grounded-object coverage, and reduces CHAIR-MSCOCO CHAIR$_i$/CHAIR$_s$ by 37.0\%/30.4\% without shortening captions.
