Search papers, labs, and topics across Lattice.
This paper investigates the vulnerabilities of chain-of-thought (CoT) monitoring in AI safety by demonstrating that model poisoning can implant backdoors in reasoning models, allowing attackers to manipulate outputs while keeping CoT traces seemingly benign. Through simple fine-tuning and a novel curriculum training approach, the authors show that models can be trained to produce specific outputs without revealing their altered behavior in reasoning traces. The findings challenge the conventional view of CoT monitoring, suggesting a need to focus on the alignment between reasoning traces and final outputs rather than merely detecting anomalies.
CoT monitoring can be easily evaded through model poisoning, allowing hidden backdoors that manipulate outputs without detection.
Chain-of-thought (CoT) monitoring is an increasingly important component of AI safety stacks but relies on the assumption that a model's reasoning trace is informative about its actions. This work studies the limits of CoT monitoring through the lens of model poisoning. We demonstrate that backdoors can be implanted into reasoning models to elicit an attacker-chosen behavior while their CoT traces appear entirely benign. We find that these CoT-Hidden backdoors can be induced through simple fine-tuning recipes across reasoning-model architectures and sizes. When direct poisoning is ineffective, we introduce a curriculum training approach that progressively teaches the model to produce an attacker-chosen output while concealing the behavior from its reasoning traces. These findings suggest that CoT monitoring may be better framed as a question about the consistency between a model's reasoning trace and its final response than as anomaly detection within a trace. We further examine the mechanisms that allow models to suppress evidence of the target behavior from their reasoning traces. Causal interventions locate a trigger-conditioned activation pathway that does not depend on the visible reasoning, and residual stream verbalizations provide an anomaly warning near answer generation, but do not identify the trigger, target, or backdoor mechanism.
