Search papers, labs, and topics across Lattice.
The paper introduces HarDBench, a benchmark to evaluate LLMs' vulnerability to draft-based co-authoring jailbreak attacks, where malicious users provide incomplete drafts prompting the LLM to generate harmful content. They find that current LLMs are highly susceptible to these attacks across domains like explosives and cyberattacks. To mitigate this, they propose a safety-utility balanced alignment approach using preference optimization, which significantly reduces harmful outputs without harming co-authoring capabilities.
LLMs are alarmingly vulnerable to jailbreak attacks when used for collaborative writing, capable of being tricked into generating harmful content from seemingly innocuous drafts.
Large language models (LLMs) are increasingly used as co-authors in collaborative writing, where users begin with rough drafts and rely on LLMs to complete, revise, and refine their content. However, this capability poses a serious safety risk: malicious users could jailbreak the models-filling incomplete drafts with dangerous content-to force them into generating harmful outputs. In this paper, we identify the vulnerability of current LLMs to such draft-based co-authoring jailbreak attacks and introduce HarDBench, a systematic benchmark designed to evaluate the robustness of LLMs against this emerging threat. HarDBench spans a range of high-risk domains-including Explosives, Drugs, Weapons, and Cyberattacks-and features prompts with realistic structure and domain-specific cues to assess the model susceptibility to harmful completions. To mitigate this risk, we introduce a safety-utility balanced alignment approach based on preference optimization, training models to refuse harmful completions while remaining helpful on benign drafts. Experimental results show that existing LLMs are highly vulnerable in co-authoring contexts and our alignment method significantly reduces harmful outputs without degrading performance on co-authoring capabilities. This presents a new paradigm for evaluating and aligning LLMs in human-LLM collaborative writing settings. Our new benchmark and dataset are available on our project page at https://github.com/untae0122/HarDBench
