Search papers, labs, and topics across Lattice.
This paper addresses the inefficiencies in Chain-of-Thought (CoT) prompting for Large Language Models (LLMs), highlighting that existing evaluators fail to penalize valid but redundant reasoning steps that inflate token usage. The authors introduce RIV-GSM8K, a benchmark designed to diagnose five types of inefficiencies, and propose CAID (Context-Aware Information Density), a novel metric that identifies low-utility reasoning steps without requiring additional training. Their empirical results demonstrate that the PACE compression strategy, informed by CAID, can reduce token consumption by 31-53% while preserving accuracy across various reasoning tasks, showcasing a significant improvement in efficiency without sacrificing performance.
Valid reasoning steps can still be inefficient, leading to a 31-53% increase in token usage without improving outcomes.
Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), yet it often incurs substantial computational costs due to over-reasoning: the generation of redundant, verbose, or irrelevant steps. While existing reasoning step evaluators effectively detect logical fallacies and factual errors, our analysis reveals a critical blind spot: they fail to penalize valid but inefficient reasoning steps that inflate token usage without contributing to the solution. To systematically diagnose this limitation, we introduce RIV-GSM8K, a diagnostic benchmark injected with five distinct types of inefficiencies, including circular reasoning and excessive decomposition. Diagnostic experiments reveal that state-of-the-art evaluators struggle to distinguish these inefficiencies from necessary reasoning. To address this gap, we propose CAID (Context-Aware Information Density), a training-free metric grounded in information theory that identifies low-utility steps. To validate the metric's practical utility, we apply it within PACE, a post-hoc compression strategy. Additional control experiments show that the gains of PACE are not explained by trivial pruning: compared with random step removal and PRM-based compression baselines, it preserves accuracy at substantially higher compression rates. Empirical results on GSM8K, StrategyQA, and ARC-Challenge demonstrate that PACE reduces token consumption by 31-53% while maintaining accuracy, confirming that CAID successfully distills informational froth from reasoning chains without compromising deductive validity.
