Search papers, labs, and topics across Lattice.
This paper introduces a training-free framework for mitigating hallucinations in large vision-language models (LVLMs) by employing Disentangled Hallucination Subspaces that capture distinct hallucination modes. Unlike traditional global editing methods, which treat all instances uniformly, this approach dynamically adjusts to the specific hallucination characteristics of each input during inference. The results show significant improvements in hallucination suppression across various benchmarks, underscoring the method's efficiency and adaptability in enhancing LVLM reliability.
Hallucinations in LVLMs can be effectively mitigated without training by leveraging per-instance subspaces that adaptively suppress specific hallucination modes.
Recent advances in large vision-language models (LVLMs) have enabled powerful multimodal reasoning by integrating visual encoders with large language models (LLMs). However, their reliability is frequently undermined by hallucinations, where generated text inaccurately describes the visual input. Although fine-tuning can mitigate this problem, it is computationally expensive and requires large, curated datasets, making training-free alternatives attractive. Among these, model editing is more promising than decoding-based approaches: decoding methods adapt outputs per input but introduce computational overhead and instability, whereas model editing modifies internal representations offline, providing a more efficient and stable solution. However, existing model-editing techniques typically rely on a single global subspace to correct hallucinations, treating all test samples identically and failing to capture diverse hallucination modes across inputs. To address this limitation, we propose a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time. Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode. During inference, the model adaptively calculates weights reflecting each input's relationship to these subspaces, guiding a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics. Extensive experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements, highlighting the robustness, generalizability, and efficiency of our approach.
