Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces QK Product Steering, a novel approach to mitigate hallucinations in vision-language models (VLMs) by editing the query-key product without requiring additional training or inference costs. By suppressing dominant singular modes in the attention mechanism, the method effectively reduces object hallucinations while maintaining the model's multimodal capabilities. Experimental results demonstrate a significant average reduction of 4.0% in hallucination rates across three GQA-based VLMs, with interpretability analyses revealing that the hallucination signal is concentrated in specific attention patterns.
Suppressing dominant attention modes can cut hallucination rates in vision-language models by 4% without any training or inference overhead.
Vision-language models (VLMs) often generate fluent but visually unsupported descriptions, especially by mentioning objects absent from the image. We propose QK Product Steering, a data-free, training-free, and zero-inference-cost weight edit for reducing object hallucination. The method directly edits the per-head query-key product, the operator that produces pre-softmax attention logits, by suppressing a small number of dominant singular modes in selected middle layers. The edited product is then mapped back to the query weights through a closed-form query-only update while keeping shared key weights fixed, making the edit compatible with grouped-query attention. We further decompose the QK product into symmetric and antisymmetric components to distinguish mutual content-similarity patterns from directional attention patterns. Across three GQA-based VLMs, QK Product Steering achieves an average relative CHAIR$_s$ reduction of $4.0\%$, while matched random-mode controls show negligible change. Interpretability ablations show that the hallucination signal is specific to dominant QK modes and is primarily localized to the symmetric mutual-attention channel. Overall, QK Product Steering offers a simple alternative to decoding-time mitigation, requiring no additional data, fine-tuning, or inference-time overhead while largely preserving general multimodal capability.