Search papers, labs, and topics across Lattice.
The paper introduces JAMA, a novel gradient-based framework that jointly optimizes textual and audio modalities to jailbreak Spoken Language Models (SLMs). By combining Greedy Coordinate Gradient (GCG) for text and Projected Gradient Descent (PGD) for audio, JAMA significantly enhances jailbreak success rates compared to unimodal attacks across various SLMs and audio types. The study also demonstrates a sequential approximation technique to accelerate the attack, highlighting the need for multimodal safety considerations in SLM development.
SLMs are shockingly vulnerable: combining adversarial audio and text unlocks 1.5x to 10x higher jailbreak rates than attacking either modality alone.
As Spoken Language Models (SLMs) integrate speech and text modalities, they inherit the safety vulnerabilities of their LLM backbone and an expanded attack surface. SLMs have been previously shown to be susceptible to jailbreaking, where adversarial prompts induce harmful responses. Yet existing attacks largely remain unimodal, optimizing either text or audio in isolation. We explore gradient-based multimodal jailbreaks by introducing JAMA (Joint Audio-text Multimodal Attack), a joint multimodal optimization framework combining Greedy Coordinate Gradient (GCG) for text and Projected Gradient Descent (PGD) for audio, to simultaneously perturb both modalities. Evaluations across four state-of-the-art SLMs and four audio types demonstrate that JAMA surpasses unimodal jailbreak rate by 1.5x to 10x. We analyze the operational dynamics of this joint attack and show that a sequential approximation method makes it 4x to 6x faster. Our findings suggest that unimodal safety is insufficient for robust SLMs. The code and data are available at https://repos.lsv.uni-saarland.de/akrishnan/multimodal-jailbreak-slm
