Search papers, labs, and topics across Lattice.
This study systematically evaluates the trade-offs associated with jailbreak defenses for large language models (LLMs), focusing on their impact on performance, over-refusal of benign inputs, and inference costs. By categorizing defenses based on operational strategies, the authors reveal that while safety gains are achieved, they often come at the expense of usability and efficiency, with rule-based defenses showing the best performance preservation. The findings highlight that many state-of-the-art defenses do not enhance downstream capabilities, providing critical insights for practitioners in selecting appropriate defenses under deployment constraints.
Rule-based defenses may protect LLMs while preserving task performance, but many popular methods compromise usability and efficiency.
Jailbreak defenses are essential for protecting large language models (LLMs), but they can also introduce secondary costs that weaken model utility. We present a systematic study of these defense trade-offs along three dimensions: performance impact, over-refusal on benign inputs, and inference cost. Rather than treating defenses as a single class, we organize them by operational strategy and examine how different strategies correlate with different side-effect profiles. Across state-of-the-art defense methods, widely used benchmark datasets, and representative open-source LLMs, we find that defenses rarely improve downstream capability, but instead vary in how they trade safety gains against usability and efficiency. In particular, rule-based defenses best preserve task performance, highly conservative self-reflective defenses often increase over-refusal, and multi-round defenses incur the largest runtime overhead. These results provide both a benchmark for evaluating defense side effects and practical guidance for selecting defenses under deployment constraints.
