Search papers, labs, and topics across Lattice.
This paper introduces Routing-based On-Policy Distillation (ROPD), a novel framework designed to enhance the safety of large language models (LLMs) by addressing vulnerabilities related to malicious data embedding. The authors demonstrate that ROPD effectively mitigates the risks associated with template mismatches, which often lead to catastrophic forgetting and performance degradation in existing safety defenses. Experimental results show that ROPD outperforms four state-of-the-art baselines, maintaining robustness and capability preservation even in the face of adversarial prompt templates.
ROPD outperforms traditional safety mechanisms by significantly reducing the risks posed by template mismatches while preserving model capabilities.
Fine-tuning is the dominant paradigm for specializing large language models (LLMs), yet it exposes a critical vulnerability: malicious data providers can embed harmful behaviors into downstream corpora, creating models that retain professional skills while violating human values on demand. Existing safety-realignment defenses often fail in practice due to three key limitations: they frequently cause catastrophic forgetting of specialized skills; their effectiveness collapses when the defender cannot observe the attacker's prompt template; and successfully realigned models remain susceptible to re-jailbreaking via simple system prompt switches. To address these challenges, we propose Routing-based On-Policy Distillation (ROPD), a novel realignment framework that models the divergence between aligned and compromised output probability distributions rather than fitting specific prompt templates. We conduct extensive experiments comparing ROPD against four state-of-the-art baselines across three datasets and three base models with varying alignment strengths. Our results demonstrate that when baseline defenses face template mismatches, often accompanied by severe degradation in downstream task performance. In contrast, ROPD substantially mitigates template-mismatch risks, maintaining superior robustness in both defense effectiveness and capability preservation. While our analysis indicates ROPD is not entirely immune to template shifts, its performance degradation is negligible compared to existing methods, establishing a new standard for robust LLM realignment.
Tsinghua AI