Search papers, labs, and topics across Lattice.
The paper introduces TRACE, a novel benchmark designed to evaluate the safety of Large Reasoning Models (LRMs) by examining prompts, reasoning traces, and final responses. This benchmark addresses the shortcomings of existing safety evaluations that primarily focus on final outputs and lack evidence annotations for safety judgments. The evaluation of 18 guardrail models on TRACE reveals that detecting unsafe content in reasoning traces is significantly more difficult than in prompts or final responses, underscoring the necessity for improved guardrail models that can accurately identify and localize unsafe content throughout the LRM inference process.
Safety judgments for reasoning traces are far more complex than for final responses, revealing critical gaps in current guardrail models' capabilities.
Large Reasoning Models (LRMs) generate intermediate reasoning traces that may contain unsafe content, even when their final responses appear safe. Guardrail models are designed to detect and block unsafe content, yet existing benchmarks for unsafe content detection focus primarily on prompts and final responses, leaving reasoning traces largely unexamined. Moreover, these benchmarks typically provide only binary safety labels, without evidence annotations that justify the judgments. To address these limitations, we introduce TRACE, an evidence-grounded safety evaluation benchmark that covers the entire LRM inference pipeline: prompts, reasoning traces, and final responses. TRACE includes prompts in two languages spanning nine risk categories and ten attack strategies. For each prompt, four LRMs generate reasoning traces and final responses, and we annotate the safety of each component and extract supporting evidence from the corresponding source text. Evaluating 18 guardrail models on TRACE reveals that safety judgment for reasoning traces is substantially more challenging than for prompts or final responses, and that current models struggle to accurately extract supporting evidence. These findings highlight the need for guardrail models that can reliably detect and precisely localize unsafe content across the LRM inference pipeline.
