Search papers, labs, and topics across Lattice.
To address persistent over-refusal on benign queries containing sensitive lexical triggers, the authors structurally decompose safety-tuning targets into boilerplate refusal statements and explanatory rationales. They find that standard refusal boilerplate actively degrades alignment by encouraging models to rely on superficial lexical cues rather than semantic intent. Crucially, fine-tuning exclusively on explanatory rationales significantly mitigates false refusals without degrading safety margins, showing strong transfer to in-context learning and compatibility with inference-time mitigations.
Boilerplate refusal prefixes like "I cannot fulfill this request" are actively sabotaging safety alignment鈥攖raining models purely on explanatory rationales slashes false refusals while preserving defensive boundaries.
Striking a balance between helpfulness and safety remains a fundamental challenge in aligning large language models. To achieve this balance, models should refuse harmful queries (e.g.,"How do I shoot someone?") while remaining responsive to benign inputs, even those superficially resembling harmful queries (e.g.,"Where can I shoot a good photo?"). However, models often struggle to distinguish genuinely harmful queries from benign queries that contain superficially risky language, resulting in false refusals. In this paper, we address the issue by decomposing a response in the safety-tuning dataset into two distinct components: (i) a boilerplate refusal statement and (ii) a rationale explaining the refusal. Our experiments and analyses show that refusal statements impede accurate discrimination between harmful and benign queries by inducing reliance on superficial cues. In contrast, training solely on rationales reduces false refusals while maintaining a comparable level of safety performance. Rationale-Only benefits also appear in our ICL configuration and remain compatible with the evaluated inference-time mitigation methods. The results emphasize the necessity of precisely curated, fine-grained safety supervision datasets and outline directions for constructing aligned agents that better reconcile helpfulness with safety.
