Search papers, labs, and topics across Lattice.
This study investigates the faithfulness of chain-of-thought (CoT) explanations in AI models, revealing that behavioral detection of unfaithfulness is significantly hindered when models provide incorrect answers. By analyzing the FaithCoT-Bench annotations, the authors find that while behavioral signals can moderately distinguish faithful reasoning from post-hoc reasoning on correct answers, they fail to detect unfaithfulness on incorrect answers, where the majority of issues reside. The findings underscore a critical gap in current auditing methods, as answer correctness is shown to be a more reliable indicator of unfaithfulness than any behavioral signal tested.
Behavioral detection of unfaithfulness in chain-of-thought reasoning falters significantly on incorrect answers, revealing a critical oversight in current auditing practices.
Chain-of-thought (CoT) explanations support oversight only if they are faithful: the stated reasoning must actually produce the answer. Auditing black-box (behavioral) detection of unfaithful CoT against FaithCoT-Bench's human annotations, we find answer correctness structures the problem at every level. Answer incorrectness alone (an oracle diagnostic, not a deployable detector) outperforms every purpose-built signal (AUROC 0.696), because 69% of annotated unfaithfulness occurs on incorrect answers. Stratifying by correctness splits detection into two regimes: on correct answers, behavioral signals moderately separate faithful from post-hoc reasoning (0.63-0.67); on incorrect answers, where most unfaithfulness lives, no tested signal is detectably above chance (replicated on all four models for benchmark-wide signals). The standard step-removal metric anti-correlates with human labels; this inversion reproduces on the benchmark's released scores and on hint-dependent counterfactually labeled traces. Linear probes decode the behaviorally blind regime in Llama-3.1-8B and the correct-answer regime in Qwen-2.5-7B, with no shared, positively aligned direction detected across regimes; instructed answer-first traces (7 models) transfer to neither annotated regime, while hint-induced unverbalized answer flips do, in model- and source-dependent settings. We also independently verify and resolve a documentation-data mismatch in the benchmark's label semantics.
