Search papers, labs, and topics across Lattice.
The paper introduces Locate-Then-Sparsify for Feature Steering (LTS-FS), a plug-and-play framework that mitigates visual hallucinations in LVLMs by applying feature steering with varying intensity across layers. They construct a synthetic dataset of hallucination cases and use causal intervention-based attribution to quantify each layer's relevance to hallucinations. LTS-FS then converts these attribution scores into layer-specific feature steering intensities, achieving targeted hallucination mitigation without sacrificing overall performance.
Stop blindly steering all layers of your LVLM - this new method uses attribution to apply targeted interventions only where hallucinations originate, preserving performance on general tasks.
Despite the significant advancements in Large Vision-Language Models (LVLMs), their tendency to generate hallucinations undermines reliability and restricts broader practical deployment. Among the hallucination mitigation methods, feature steering emerges as a promising approach that reduces erroneous outputs in LVLMs without increasing inference costs. However, current methods apply uniform feature steering across all layers. This heuristic strategy ignores inter-layer differences, potentially disrupting layers unrelated to hallucinations and ultimately leading to performance degradation on general tasks. In this paper, we propose a plug-and-play framework called Locate-Then-Sparsify for Feature Steering (LTS-FS), which controls the steering intensity according to the hallucination relevance of each layer. We first construct a synthetic dataset comprising token-level and sentence-level hallucination cases. Based on this dataset, we introduce an attribution method based on causal interventions to quantify the hallucination relevance of each layer. With the attribution scores across layers, we propose a layerwise strategy that converts these scores into feature steering intensities for individual layers, enabling more precise adjustments specifically on hallucination-relevant layers. Extensive experiments across multiple LVLMs and benchmarks demonstrate that our LTS-FS framework effectively mitigates hallucination while preserving strong performance.
