Search papers, labs, and topics across Lattice.
The paper introduces Token-level data selection for Safe LLM fine-tuning (TOSS), a framework to mitigate safety degradation during LLM fine-tuning by quantifying the safety risk of individual tokens based on loss differences between a safety-degraded model and a utility-oriented model. TOSS identifies and removes unsafe tokens at a finer granularity than sample-level methods, preserving task-specific information and improving the safety-utility trade-off. The authors further propose TOSS-Pro, a progressive refinement strategy that iteratively improves the safety-degraded model's ability to detect unsafe tokens, leading to superior performance compared to existing methods.
Fine-tuning LLMs doesn't have to compromise safety: TOSS identifies and removes unsafe tokens with greater precision than sample-level methods, preserving utility.
Fine-tuning large language models (LLMs) on custom datasets has become a standard approach for adapting these models to specific domains and applications. However, recent studies have shown that such fine-tuning can lead to significant degradation in the model's safety. Existing defense methods operate at the sample level and often suffer from an unsatisfactory trade-off between safety and utility. To address this limitation, we perform a systematic token-level diagnosis of safety degradation during fine-tuning. Based on this, we propose token-level data selection for safe LLM fine-tuning (TOSS), a novel framework that quantifies the safety risk of each token by measuring the loss difference between a safety-degraded model and a utility-oriented model. This token-level granularity enables accurate identification and removal of unsafe tokens, thereby preserving valuable task-specific information. In addition, we introduce a progressive refinement strategy, TOSS-Pro, which iteratively enhances the safety-degraded model's ability to identify unsafe tokens. Extensive experiments demonstrate that our approach robustly safeguards LLMs during fine-tuning while achieving superior downstream task performance, significantly outperforming existing sample-level defense methods. Our code is available at https://github.com/Polly-LYP/TOSS.
