Search papers, labs, and topics across Lattice.
This paper investigates the vulnerability of Vision-Language Models (VLMs) to relation hallucination under visual perturbations like rotation and noise. Experiments across models and datasets reveal that even slight distortions substantially impair relational reasoning. While prompt-based augmentation and image preprocessing techniques offer some mitigation, they don't eliminate the hallucination, indicating a fundamental weakness in current VLMs.
VLMs can be easily tricked into "hallucinating" object relationships with simple image rotations or noise, revealing a surprising fragility in their multimodal reasoning.
Vision-language models (VLMs) achieve strong multimodal performance but remain prone to relation hallucination, which requires accurate reasoning over inter-object interactions. We study the impact of visual perturbations, specifically rotation and noise, and show that even mild distortions significantly degrade relational reasoning across models and datasets. We further evaluate prompt-based augmentation and preprocessing strategies (orientation correction and denoising), finding that while they offer partial improvements, they do not fully resolve hallucinations. Our results reveal a gap between perceptual robustness and relational understanding, highlighting the need for more robust, geometry-aware VLMs.
