Search papers, labs, and topics across Lattice.
This paper introduces SafeSteer, a novel approach to aligning Large Language Models (LLMs) with human values through localized on-policy distillation focused on safety tokens. By constructing a safety teacher via activation steering and implementing a token selection algorithm, SafeSteer effectively minimizes the alignment tax while preserving general capabilities. Experimental results demonstrate that SafeSteer outperforms existing methods, achieving strong safety performance across seven benchmarks with minimal degradation on general capabilities, using significantly fewer harmful samples than previous approaches.
Achieving superior safety alignment with LLMs using only 100 harmful samples, SafeSteer drastically cuts alignment costs while maintaining model performance.
Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive general-purpose data or auxiliary reward models. In this paper, we argue that, because safety features are inherently sparse within the output distribution, alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performs on-policy distillation confined to safety tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety token selection algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these tokens during training to preserve general capabilities. Experimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety benchmarks with only minimal degradation on five general capability benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing alignment cost. More details are on our project page at https://anjingkun.github.io/SafeSteer.
