Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the relationship between the legibility of reasoning traces in chain-of-thought models and their actual importance in decision-making. By operationalizing the importance of reasoning steps through Monte Carlo rollouts, the authors assess whether LLM judges can accurately identify high-advantage steps. The results reveal that while capable LLMs can outperform a baseline in recognizing important steps, they still fall short of fully capturing the underlying importance, indicating a significant gap between legibility and interpretability in model reasoning.
LLM judges may recognize important reasoning steps, but they still miss critical nuances, highlighting a gap between legibility and true interpretability in AI reasoning.
Reasoning traces from chain-of-thought models appear to offer a legible window into how a model arrives at its answer. A growing body of work treats them as such, using LLM judges to diagnose errors, evaluate faithfulness, and provide step-level supervision via process reward models and generative critics. These practices rely on the text of a reasoning step carrying information about its functional role. But does the text actually encode information about which reasoning steps matter? We operationalize the importance of a reasoning step as its advantage: the change in expected reward, e.g., producing the correct final answer, from including that step, estimated via Monte Carlo rollouts. Basing ground truth on these estimates, we evaluate whether LLM judges can identify high-advantage steps and find that sufficiently capable LLMs can outperform a prevalence baseline but fall well short of a noise ceiling. Fine-tuning a model as a step-level critic yields strong improvement for incorrect responses but remains distant from ceiling for correct responses, suggesting that step importance is only partially recoverable from the text of the reasoning trace. Our findings contribute to a growing body of chain-of-thought faithfulness work that cautions against treating the legibility of reasoning traces as interpretability, especially with implications for process reward modeling.