Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Barrier-Regulated Adaptive Closed-form Steering (BRACS), a training-free inference method to mitigate object hallucination in LVLMs by monitoring attention to measure visual grounding and applying closed-form corrective updates to hidden states only when grounding deteriorates. BRACS addresses limitations of existing methods by incorporating an explicit grounding objective and adaptively adjusting correction strength based on grounding failure. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat demonstrate that BRACS outperforms prior methods on hallucination benchmarks while maintaining or improving performance on general multimodal tasks and achieving comparable or better efficiency.
LVLMs can be steered away from hallucinating objects with a training-free method that monitors attention and applies closed-form corrections only when visual grounding weakens, improving accuracy and efficiency.
Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image, largely because visual grounding weakens as decoding progresses. Existing inference-time mitigation methods modify logits or hidden states throughout generation, but they suffer from three key limitations: they lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure. We propose BRACS (Barrier-Regulated Adaptive Closed-form Steering), a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering. BRACS monitors the model's own attention to measure visual grounding and applies corrections to the hidden states only when grounding deteriorates. The corrective update is computed analytically in closed form, requiring no training of auxiliary networks or model retraining. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat show that BRACS consistently outperforms prior methods on hallucination benchmarks, reducing CHAIR$_s$ by 9.4 points and improving POPE F1 by 2.7 points, while matching or improving performance on four general multimodal benchmarks. BRACS also remains efficient, operating at 80% of greedy decoding throughput and achieving 1.3 times higher speed on average than the baselines.