Search papers, labs, and topics across Lattice.
MUSE is introduced as an open-source, run-centric platform for multimodal safety evaluation of LLMs, incorporating cross-modal payload generation, multi-turn attack algorithms, model routing, and an LLM judge with a five-level safety taxonomy. The platform introduces Inter-Turn Modality Switching (ITMS) to augment multi-turn attacks with per-turn modality rotation, revealing vulnerabilities in cross-modal alignment. Experiments on six multimodal LLMs demonstrate that multi-turn strategies can achieve up to 90-100% Attack Success Rate (ASR), and that modality effects are model-family-specific, highlighting the need for provider-aware cross-modal safety testing.
Multimodal LLMs that ace single-turn safety can still be tricked into 90-100% ASR with multi-turn attacks, especially when modalities switch mid-conversation.
Safety evaluation and red-teaming of large language models remain predominantly text-centric, and existing frameworks lack the infrastructure to systematically test whether alignment generalizes to audio, image, and video inputs. We present MUSE (Multimodal Unified Safety Evaluation), an open-source, run-centric platform that integrates automatic cross-modal payload generation, three multi-turn attack algorithms (Crescendo, PAIR, Violent Durian), provider-agnostic model routing, and an LLM judge with a five-level safety taxonomy into a single browser-based system. A dual-metric framework distinguishes hard Attack Success Rate (Compliance only) from soft ASR (including Partial Compliance), capturing partial information leakage that binary metrics miss. To probe whether alignment generalizes across modality boundaries, we introduce Inter-Turn Modality Switching (ITMS), which augments multi-turn attacks with per-turn modality rotation. Experiments across six multimodal LLMs from four providers show that multi-turn strategies can achieve up to 90-100% ASR against models with near-perfect single-turn refusal. ITMS does not uniformly raise final ASR on already-saturated baselines, but accelerates convergence by destabilizing early-turn defenses, and ablation reveals that the direction of modality effects is model-family-specific rather than universal, underscoring the need for provider-aware cross-modal safety testing.
