Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper extends the RESTA defense, originally designed for language models, to vision-language models (VLMs) to improve their robustness against jailbreaking attacks. The authors evaluate RESTA's performance on the JailBreakV-28K benchmark and find it effectively reduces attack success rates. A key finding is that aligning the injected noise with the original token embedding vectors (directional embedding noise) further enhances RESTA's defensive capabilities.
Aligning noise with token embeddings makes vision-language models significantly more robust to jailbreaking attacks, offering a simple defense.
The safety and reliability of vision-language models (VLMs) are a crucial part of deploying trustworthy agentic AI systems. However, VLMs remain vulnerable to jailbreaking attacks that undermine their safety alignment to yield harmful outputs. In this work, we extend the Randomized Embedding Smoothing and Token Aggregation (RESTA) defense to VLMs and evaluate its performance against the JailBreakV-28K benchmark of multi-modal jailbreaking attacks. We find that RESTA is effective in reducing attack success rate over this diverse corpus of attacks, in particular, when employing directional embedding noise, where the injected noise is aligned with the original token embedding vectors. Our results demonstrate that RESTA can contribute to securing VLMs within agentic systems, as a lightweight, inference-time defense layer of an overall security framework.