Search papers, labs, and topics across Lattice.
This study investigates the faithfulness of latent reasoning methods during training by analyzing how their causal contributions to final answers evolve across checkpoints. By employing counterfactual edits and noise-ablation patches, the authors reveal that latent reasoning can appear unfaithful at convergence while exhibiting divergent training trajectories. Key findings indicate that the causal influence of latent reasoning steps diminishes over time, particularly for binary choices, emphasizing the complex relationship between training dynamics and reasoning fidelity.
Latent reasoning methods may seem unfaithful at convergence, but their causal contributions to answers decay over training, revealing hidden complexities in model behavior.
Latent reasoning methods perform multi-step inference entirely in the model's continuous hidden states, promising more compact and efficient reasoning. However, these opaque hidden states raise a question of faithfulness: whether these latent reasoning steps causally drive the final answer. Prior work investigates this question at converged checkpoints and reports several unfaithful behaviors, such as latent reasoning steps that can be replaced without changing the answer, but leaves how these behaviors form during training unexamined. We instead track how faithfulness evolves across saved checkpoints for different latent reasoning paradigms, applying a verifiable counterfactual edit on the input and a noise-ablation activation patch on the latent reasoning steps. We find that (i) at the output level, latent reasoning methods can look similarly unfaithful at convergence under counterfactual edits while following qualitatively divergent trajectories; (ii) at the activation level, the causal contribution of latent reasoning steps to the final answer decays across training for both paradigms, with the examples that flip on the output side in (i) also being the examples on which this contribution decays; and (iii) the activation-level trajectory diverges by answer format, decaying on binary choice and rising on open-ended decoding. These findings highlight that latent reasoning faithfulness depends on training stage and answer format.
