Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel method called MIST for compressing chain-of-thought (CoT) reasoning by evaluating token-level saliency from a model-internal perspective. By measuring the necessity and sufficiency of each token's contribution to the answer computation, MIST generates a unified importance score that effectively prunes reasoning chains. The results demonstrate that MIST outperforms existing baseline methods across multiple reasoning benchmarks, significantly reducing inference costs while maintaining performance.
Token-level saliency derived from a model's internal reasoning process can drastically improve the efficiency of chain-of-thought compression, outperforming traditional methods.
Chain-of-thought (CoT) reasoning improves multi-step problem solving, but long reasoning traces inflate inference cost. Token-level CoT compression reduces this cost by pruning full reasoning chains into shorter traces for model adaptation, making token selection the central challenge. Existing methods often rely on external scorers or heuristic signals only indirectly tied to the model's internal answer computation. We instead adopt a model-internal perspective: as the model forms an answer, each reasoning token leaves a ripple in the residual stream, the model's \emph{stream of thought}, and the magnitude of this ripple reflects the token's contribution to the answer computation. Building on this view, we propose \textsc{MIST} (Model-Internal Saliency for Token-level CoT compression), which defines token importance along two complementary axes: \emph{necessity}, the drop in answer likelihood when a token's internal contribution is removed, and \emph{sufficiency}, the gain in answer likelihood when that contribution alone is provided. Combining the two yields a unified importance score for pruning. Across four reasoning benchmarks and four models, \textsc{MIST} consistently outperforms baseline methods, suggesting that model-internal saliency provides an effective proxy for reasoning-token importance.