Search papers, labs, and topics across Lattice.
The paper introduces STAR-1, a safety dataset of only 1,000 examples designed for aligning large reasoning models (LRMs) by focusing on diversity, deliberative reasoning, and rigorous GPT-4o-based filtering. Fine-tuning LRMs with STAR-1 achieves a 40% average improvement in safety across four benchmarks, with minimal (1.1%) regression in reasoning ability across five reasoning tasks. Ablation studies validate the design principles and efficacy of STAR-1 across both LRMs and traditional LLMs.
Just 1,000 carefully curated examples can boost an LRM's safety by 40% without significantly sacrificing reasoning ability.
This paper introduces STAR-1, a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) like DeepSeek-R1. Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs. Specifically, we begin by integrating existing open-source safety datasets from diverse sources. Then, we curate safety policies to generate policy-grounded deliberative reasoning samples. Lastly, we apply a GPT-4o-based safety scoring system to select training examples aligned with best practices. Experimental results show that fine-tuning LRMs with STAR-1 leads to an average 40% improvement in safety performance across four benchmarks, while only incurring a marginal decrease (e.g., an average of 1.1%) in reasoning ability measured across five reasoning tasks. Extensive ablation studies further validate the importance of our design principles in constructing STAR-1 and analyze its efficacy across both LRMs and traditional LLMs. Our project page is https://ucsc-vlaa.github.io/STAR-1.
Mila