Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors evaluate the mechanistic validity of chain-of-thought reasoning by leveraging iGSM, a synthetic math benchmark where step-by-step arithmetic and semantic variable dependencies are programmatically verifiable. While answer correctness mirrors trace validity in-distribution, the two decouple out-of-distribution, with 31.6% of correct answers on hard problems executing semantically invalid computations despite passing surface-level arithmetic checks. Furthermore, training interventions show that models maintain near-clean accuracy even when trained on shuffled or swapped traces, indicating that scratchpads do not reliably reflect actual computational paths.
Up to 31.6% of correct model solutions rely on semantically invalid reasoning traces that look deceptively sound, revealing that chain-of-thought generation frequently acts as an unfaithful facade rather than a true computation graph.
Chain-of-thought traces are widely read as records of how models reach their answers, informing debugging, agent auditing, and claims about reasoning. Testing this interpretation is difficult because natural-language thinking traces are rarely mechanically verifiable. We revisit it in iGSM, a synthetic grade-school mathematics benchmark designed to study thinking traces and used to support claims of learned reasoning and planning. Crucially, iGSM exposes the exact quantities and dependencies that a correct solution should use, allowing generated traces to be checked programmatically step by step and enabling us to test whether correct answers are reliably accompanied by valid traces. We first evaluate models trained exclusively on valid, minimal traces. Answer correctness and trace validity nearly coincide in distribution but decouple out of distribution: on the hardest instances, 31.6% of correct answers have invalid traces, over half of which pass all syntactic and arithmetic checks but fail semantic dependency checks. We then intervene on trace supervision. Non-minimal training traces induce non-minimal outputs, while re-asking the same problem with a different query reveals computations inherited from the original query, weakening minimality as evidence of selective planning. Shuffling tokens in 10% of training trace sentences preserves near-clean accuracy even out of distribution despite no trace passing verification. Swapped training traces likewise retain high in-distribution accuracy. We discuss the implications of these findings for chain-of-thought monitoring and interpretation in the context of AI safety.