Search papers, labs, and topics across Lattice.
This paper introduces DataRx, a missingness-aware sampling method designed to enhance the safety of large language models (LLMs) during task-specific fine-tuning. By leveraging high-dimensional hidden representations to identify and fill gaps in safety capabilities, DataRx significantly reduces the attack success rate of Llama3-8B-Instruct from 59.23% to 13.70% with just a 1% increase in safety samples. The findings underscore the importance of targeted safety data selection over random mixing, paving the way for improved safety in LLM applications.
Targeted safety sampling can slash attack success rates in fine-tuned LLMs from over 59% to under 14% with minimal additional data.
Task-specific fine-tuning can improve the performance of large language models (LLMs) on downstream tasks. However, our study reveals that task-specific fine-tuning can also weaken the safety guardrails of aligned LLMs. A widely adopted strategy for preserving safety during fine-tuning is to incorporate safety data. Although previous studies have shown that randomly mixing safety data can alleviate safety degradation, the underlying principle determining why some safety examples are more effective than others still remains unclear. In this paper, we propose DataRx, a missingness-aware sampling method for selecting safety-critical examples. DataRx is based on the hypothesis that a safety sample is more effective when the selected examples provide safety signals that fill the missing parts of LLMs'safety capabilities. DataRx's key insight is leveraging high-dimensional hidden representations rather than discrete tokens to quantify the safety signal gap between the target model's native response and the safety reference response. The results show that, with only 1% additional safety samples from BeaverTails, DataRx reduces the average attack success rate of Llama3-8B-Instruct across seven downstream tasks from 59.23% under random sampling to 13.70%. In addition, DataRx can be combined with the existing safety data synthesis method to further enhance safety defenses during fine-tuning. We hope that DataRx will inspire more data-centric defense research.
