Search papers, labs, and topics across Lattice.
This paper introduces Phase-wise Self-Reward Decoding (PSRD), a novel inference-time framework to mitigate visual hallucination in Large Vision-Language Models (LVLMs) without external supervision. PSRD leverages the observation that hallucination exhibits phase-wise dynamic patterns, peaking at the onset of each semantic phase, to guide online hallucination correction. By distilling hallucination guidance signals into a lightweight reward model, PSRD achieves a 50% reduction in hallucination rate on LLaVA-1.5-7B and outperforms existing post-hoc methods across multiple benchmarks.
LVLMs hallucinate in predictable bursts, and this self-rewarding decoding strategy slashes those errors in half.
Large Vision-Language Models (LVLMs) still struggle with vision hallucination, where generated responses are inconsistent with the visual input. Existing methods either rely on large-scale annotated data for fine-tuning, which incurs massive computational overhead, or employ static post-hoc strategies that overlook the dynamic nature of hallucination emergence. To address these, we introduce a new self-rewarding framework, enabling dynamic hallucination mitigation at inference time without external supervision. On the empirical side, we reveal that visual hallucination exhibits phase-wise dynamic patterns, peaking at the onset of each semantic phase. Drawing on these insights, we propose \textbf{PSRD} (\textbf{Phase-wise \textbf{S}elf-\textbf{R}eward \textbf{D}ecoding) for online hallucination correction guided by phase-wise self-reward signals. To reduce the cost of repeated self-evaluation during decoding, we distill the hallucination guidance signal from LVLMs into a lightweight reward model. The reward model subsequently provides on-the-fly guidance for targeted intervention during the decoding process, enabling precise hallucination suppression. The proposed PSRD significantly reduces the hallucination rate of LLaVA-1.5-7B by 50.0% and consistently outperforms existing post-hoc methods across five hallucination evaluation benchmarks for four LVLMs. Further analysis confirms that PSRD effectively mitigates hallucination propagation and achieves a highly controllable trade-off between strong performance and inference efficiency.
