Search papers, labs, and topics across Lattice.
This paper introduces ReWEIGH, a decoding intervention designed to mitigate hallucinations in large vision-language models (LVLMs) by calibrating token-level ordinal visual evidence. By leveraging the model's visual-token states and employing a rank-based pooling method, ReWEIGH effectively assesses the support an image provides for each token candidate, applying a penalty to those that fall below a reference threshold. The approach demonstrates a significant reduction in hallucinated object mentions by up to 21.3% across various model architectures, while maintaining or enhancing overall performance with minimal added latency.
Hallucinations in LVLMs can be reduced by over 21% without sacrificing performance, thanks to a novel method of calibrating visual evidence during decoding.
Large vision-language models (LVLMs) often hallucinate, generating content that the input image does not support. Preventing such content during decoding calls for a candidate-specific measure of how strongly the image supports the token under consideration. The model's visual-token states offer a natural source of this evidence because projecting each state through the output head reveals which vocabulary items that position favors. These position-wise readouts cannot be pooled directly because their probability magnitudes are not comparable across visual positions. Vocabulary ranks provide a scale-invariant basis for pooling, but tokens still differ systematically in their typical rank-based evidence. We propose ReWEIGH, a training-free decoding intervention that aggregates these ranks across visual positions and compares each candidate with a token-specific reference estimated from unlabeled images. At inference, ReWEIGH caches the image evidence during prefill and applies a bounded penalty only to candidates that fall below their reference. On four 7B backbones, ReWEIGH reduces hallucinated object mentions by up to 21.3% while largely preserving or improving descriptive and general performance. With evidence cached, the average added latency is 1.33% per token, and the reductions extend across six architecture families to 32B parameters.
