Search papers, labs, and topics across Lattice.
This paper introduces RVSD, a novel decoding framework that integrates token sparsification with Semantic-Space Visual Retrieval (SSVR) to address the issue of Visual Hallucinations (VHs) in large vision-language models. By employing a semantics-directed token selection strategy, RVSD effectively reduces redundancy while ensuring the retention of essential visual information, all without the need for additional training or curated datasets. Experimental results show that RVSD not only mitigates VHs but also excels in long-context generation scenarios, outperforming existing methods.
Visual Hallucinations in large vision-language models can be significantly reduced without the overhead of additional training or curated datasets, thanks to RVSD's innovative approach.
Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose \textbf{RVSD} (\underline{R}etrieval \underline{V}ision \underline{S}parse \underline{D}ecoding), a training-free and plug-and-play decoding framework that, for the first time, unifies token sparsification and \textbf{Semantic-Space Visual Retrieval} (SSVR) within a single decoding pass. Within RVSD, we introduce a \textbf{semantics-directed token selection} strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.\footnote{https://github.com/canjie-liu/RVSD}
