Search papers, labs, and topics across Lattice.
The paper investigates the sensitivity of Vision-Language Models (VLMs) to semantic cues in safety judgments. They introduce a semantic steering framework that uses controlled textual, visual, and cognitive interventions to influence VLM safety decisions without altering the underlying scene. Through the SAVeS benchmark, they find that VLMs heavily rely on learned visual-linguistic associations, making their safety decisions highly susceptible to semantic cues rather than grounded visual understanding.
VLMs' safety judgments are easily manipulated by simple semantic cues, revealing a reliance on superficial associations rather than true visual understanding.
Vision-language models (VLMs) are increasingly deployed in real-world and embodied settings where safety decisions depend on visual context. However, it remains unclear which visual evidence drives these judgments. We study whether multimodal safety behavior in VLMs can be steered by simple semantic cues. We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content. To evaluate these effects, we propose SAVeS, a benchmark for situational safety under semantic cues, together with an evaluation protocol that separates behavioral refusal, grounded safety reasoning, and false refusals. Experiments across multiple VLMs and an additional state-of-the-art benchmark show that safety decisions are highly sensitive to semantic cues, indicating reliance on learned visual-linguistic associations rather than grounded visual understanding. We further demonstrate that automated steering pipelines can exploit these mechanisms, highlighting a potential vulnerability in multimodal safety systems.
