Search papers, labs, and topics across Lattice.
This paper investigates the cross-lingual transferability of jailbreak detection using semantic similarity against a fixed English codebook of jailbreak prompts, without retraining or language-specific adaptation. They evaluate this approach across four languages, multiple translation pipelines, safety benchmarks, embedding models, and target LLMs (Qwen, Llama, GPT-3.5). Results show high separability (AUC up to 0.99) on curated benchmarks but significant performance degradation (AUC ≈ 0.60-0.70) under distribution shift to behaviorally diverse unsafe prompts, highlighting limitations in generalizing to heterogeneous jailbreak attempts.
Jailbreak defenses relying on semantic similarity crumble when faced with diverse, real-world multilingual attacks, even if they ace the textbook examples.
Safety mechanisms for large language models (LLMs) remain predominantly English-centric, creating systematic vulnerabilities in multilingual deployment. Prior work shows that translating malicious prompts into other languages can substantially increase jailbreak success rates, exposing a structural cross-lingual security gap. We investigate whether such attacks can be mitigated through language-agnostic semantic similarity without retraining or language-specific adaptation. Our approach compares multilingual query embeddings against a fixed English codebook of jailbreak prompts, operating as a training-free external guardrail for black-box LLMs. We conduct a systematic evaluation across four languages, two translation pipelines, four safety benchmarks, three embedding models, and three target LLMs (Qwen, Llama, GPT-3.5). Our results reveal two distinct regimes of cross-lingual transfer. On curated benchmarks containing canonical jailbreak templates, semantic similarity generalizes reliably across languages, achieving near-perfect separability (AUC up to 0.99) and substantial reductions in absolute attack success rates under strict low-false-positive constraints. However, under distribution shift - on behaviorally diverse and heterogeneous unsafe benchmarks - separability degrades markedly (AUC $\approx$ 0.60-0.70), and recall in the security-critical low-FPR regime drops across all embedding models.
