Search papers, labs, and topics across Lattice.
This study investigates the dynamics of chain-of-thought (CoT) reasoning in large language models (LLMs) to identify and correct reasoning failures by analyzing how the reasoning process evolves over time. By examining LLM performance on Boolean satisfiability tasks, the authors reveal that failures often manifest as premature verification and incorrect assumptions about satisfiability, rather than isolated errors in individual reasoning steps. A targeted intervention significantly improves accuracy, demonstrating that understanding CoT dynamics can enhance both diagnostics and correction of reasoning errors in LLMs.
Premature verification and flawed assumptions in reasoning traces can lead to dramatic drops in LLM accuracy, but targeted interventions can recover performance by over 70%.
Chain-of-thought (CoT) reasoning improves large language model (LLM) performance while also providing an observable interface to the model's reasoning process. Existing approaches that leverage verbalized CoTs to monitor reasoning correctness, however, largely evaluate the semantic correctness or consistency of individual intermediate steps, rather than how the reasoning process evolves across the trace. As a result, failures distributed across the reasoning trajectory, rather than those localized to a single incorrect step, remain comparatively underexplored. Furthermore, verbalized CoTs need not faithfully reflect the model's internal reasoning, motivating analyses that do not treat individual statements as literal accounts of internal computation. In this work, we therefore ask whether the dynamics of visible CoT can be leveraged to systematically distinguish successful from failed reasoning without assuming such semantic faithfulness. We study a range of LLMs on verifiable Boolean satisfiability tasks with variable complexity, enabling controlled comparisons near each model's capability frontier. Tagging CoT sentences by reasoning function reveals premature verification collapse on SAT problems: incorrect traces enter clause checking earlier, repeat similar operations, and finalize sooner. On UNSAT problems, models presumptuously move towards incorrect SAT conclusions, checking candidate assignments rather than deriving contradictions across constructed cases. Subsequently, a targeted proof-search prompt intervention raises Llama3-70B accuracy from 13.3% to 85%, correcting 84.6% of these errors. These results show that capability failures can manifest as distributed, task-dependent changes in the structure of visible reasoning, and that CoT dynamics agnostic to whether the verbalized trace reflects the model's internal computations can help diagnose and correct failures.
