Search papers, labs, and topics across Lattice.
This paper investigates the safety risks associated with merging fine-tuned models, revealing that even individually aligned models can pose significant vulnerabilities due to their shared pretrained foundations. The authors introduce a novel threat model where attackers can exploit these vulnerabilities by generating generalized jailbreak prompts that work across merged models. Their proposed method, Basin-Aware Jailbreak (BAJ), employs a min-max optimization approach to create transferable adversarial suffixes, demonstrating high success rates in breaching model safety across various configurations.
Merging aligned models can inadvertently expose them to jailbreak attacks rooted in their shared pretrained foundations, with a new method that exploits this vulnerability showing high transfer success rates.
Model merging enables combining multiple fine-tuned models without additional training, but its safety implications remain poorly understood. Prior work primarily attributes merging risks to unsafe constituent models, implicitly assuming that merging individually aligned models preserves safety. In contrast, we show that model merging reveals a previously overlooked jailbreak risk rooted in the pretrained foundation model, even when all constituent models are individually safety-aligned. Motivated by this observation, we study a new threat setting where an attacker constructs jailbreak prompts that generalize across merged models sharing the same pretrained backbone, without access to the exact merging coefficients or constituent checkpoints. To exploit this phenomenon, we propose \textbf{Basin-Aware Jailbreak (BAJ)}, which formulates jailbreak generation as a min--max optimization over the merging space to produce transferable adversarial suffixes across merged model families. Experiments across diverse backbones and merging settings show that BAJ achieves consistently high transfer success rates and remains effective under existing defenses.
