Search papers, labs, and topics across Lattice.
This study investigates attribute hallucination in vision-language models (VLMs), where models accurately identify objects but mischaracterize their properties. By introducing VISOR, a framework that combines null-image diagnostics with routed remediation, the authors reveal that visual signals are the primary source of false positives, while language-prior signals contribute minimally. The framework effectively addresses two distinct failure modes in attribute prediction and significantly reduces false positives across multiple VLM families without relying on the previously dominant language-prior explanation.
Visual signals, not language priors, drive attribute hallucination in VLMs, leading to a novel framework that effectively mitigates this issue.
Attribute hallucination---where vision-language models (VLMs) correctly identify an object but mischaracterize its properties---is prevalent yet mechanistically poorly understood. The dominant explanation, language-prior dominance, has motivated prior-suppression methods, but this explanation has not been directly tested at the attribute level. We present VISOR (Visual-Operational Remediation), a unified framework that couples null-image-based diagnosis with routed remediation. Its VSNR diagnostic decomposes each prediction into a visual logit signal and a language-prior signal. Across 10,791 negative-ground-truth samples from three VLM families and three attribute types, the visual signal strongly predicts false positives, whereas the language-prior signal is near chance. VISOR uses this diagnosis to separate two failure modes: low-margin but directionally correct visual signals in color/state attributes, and low-SNR or misaligned visual signals in material attributes. The same diagnosis routes each query to the appropriate operator: calibration for threshold-placement errors, abstention for training-free low-SNR handling, or targeted visual adaptation for material failures that prior suppression cannot correct. Across Qwen, InternVL, and LLaVA, VISOR reduces attribute false positives without relying on the prior-dominance assumption.
