Search papers, labs, and topics across Lattice.
This paper introduces SafeRI, a novel framework for on-demand safety intervention in vision-language models (VLMs) that avoids the pitfalls of global safety modifications. By employing a streaming recognition mechanism alongside a gated LoRA approach, SafeRI dynamically assesses the safety of the model's pre-token generation state and selectively intervenes to redirect unsafe outputs. Experimental results across various benchmarks reveal that this method effectively maintains multimodal capabilities while enhancing safety, outperforming traditional always-on safety interventions.
On-demand safety interventions can significantly improve the performance of vision-language models without sacrificing their multimodal reasoning capabilities.
Existing safety alignment methods for vision-language models usually modify the model behavior globally: once the safety parameters are trained or loaded, they participate in both unsafe and already-safe generations. This always-on intervention can unnecessarily perturb the model's original reasoning path and degrade general multimodal capabilities. We argue that safety alignment should be an on-demand intervention rather than a permanent modification to every decoding trajectory. To this end, we propose a streaming recognition and gated LoRA framework for intrinsic VLM safety. During autoregressive generation, a lightweight recognizer estimates whether the current pre-token generation state is safe or unsafe. Its output updates the LoRA gate for the following decoding step; otherwise, generation follows the frozen-backbone policy. The LoRA module is trained from unsafe prefixes, transition statements, and safe continuations, so that it learns to redirect unsafe generations back to safe responses after activation. Experiments across multiple safety and general-purpose benchmarks demonstrate the effectiveness of our method in post-alignment settings.
