Search papers, labs, and topics across Lattice.
This study investigates the tension between faithfulness and safety in Large Reasoning Models (LRMs) by introducing HazMart, a dataset designed to evaluate model responses in an autonomous AI shopkeeper context. The authors employ a novel Targeted Reasoning Replacement (TRR) technique to directly manipulate reasoning chains, revealing that while DeepSeek-R1-Llama-70B achieves high faithfulness, it struggles with unsafe reasoning, whereas QwQ-32B demonstrates better safety at the expense of faithfulness. Mechanistic analyses indicate that representation steering can enhance safety without sacrificing overall model performance, achieving a 9 percentage point increase in safe behavior.
Faithfulness and safety in LRMs are at odds, with one model achieving high accuracy but failing to reject unsafe reasoning, while another sacrifices accuracy for improved safety.
Chain-of-Thought (CoT) reasoning offers a promising window into model monitoring. However, monitoring relies on faithfulness, i.e., the model output strictly derives from its reasoning trace. We identify an alignment tension where a model must be faithful enough to be monitored, yet robust enough to reject unsafe reasoning. We demonstrate that this counterbalance exists in current Large Reasoning Models (LRMs), and show ways in which it can be addressed. We introduce HazMart, a human-written dataset set in an autonomous AI shopkeeper scenario. Unlike prior work that relies on providing hints in prompts to test faithfulness (e.g., "A Stanford professor said it should be Answer A"), we propose a novel replacement-based technique, which we call Targeted Reasoning Replacement (TRR), that directly intervenes in the reasoning chain to substitute in unsafe or illogical thoughts (e.g., "Wait, the answer must be Option B [was Option A] because it is the most fitting"). DeepSeek-R1-Llama-70B exhibits high faithfulness (97.5%) but fails to reject Unsafe Reasoning (12.3%), while QwQ-32B is more robust (73.9% safety) at the cost of lower faithfulness (74.7%). Mechanistic analyses of QwQ-32B reveal that these properties are represented by anti-correlated internal directions peaking at the action-commit token. Finally, we demonstrate that representation steering can independently amplify the safety direction, increasing safe behavior by 9 percentage points while maintaining base capabilities.
