Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Post-training quantization (PTQ) degrades reasoning performance and increases overthinking in LLMs, diminishing efficiency gains. RATIO addresses this by identifying model-specific overthinking tokens via Quantization-aware Reasoning Behavior Analysis (QRBA) and assigning tailored penalties using Token-Specific Penalty Determination (TSPD) guided by full-precision models. Experiments demonstrate RATIO significantly improves accuracy and reduces CoT length compared to existing methods, offering a superior accuracy-efficiency trade-off.
Quantized reasoning models overthink more, negating efficiency gains, but a new method precisely targets and penalizes these "overthinking" tokens without retraining.
Post-training quantization (PTQ) has become a widely adopted technique for reducing the memory footprint and inference cost of large language models (LLMs). However, recent studies reveal that when applied to reasoning models, PTQ not only degrades reasoning performance but also exacerbates overthinking, leading to longer reasoning trajectories. These issues may offset the efficiency gains expected from lower-precision inference. Existing approaches mainly rely on complex optimization procedures. More recent lightweight inference strategies instead use predefined overthinking markers, limiting their adaptability across quantized models. To address these issues, we propose Reasoning Analysis and Token-level Inference Optimization (RATIO), a framework that identifies model-specific overthinking tokens and assigns each a tailored penalty. RATIO first introduces Quantization-aware Reasoning Behavior Analysis (QRBA) to identify overthinking tokens by analyzing discrepancies between full-precision and quantized models. It then adopts Token-Specific Penalty Determination (TSPD), which leverages full-precision guidance to derive token-specific penalties without additional training. Extensive experiments show that RATIO achieves a better accuracy-efficiency trade-off than existing token-level interventions. Specifically, RATIO achieves up to 9.8 points accuracy improvement and reduces chain-of-thought (CoT) length by up to 51.3% compared with quantized baselines. The code will be available at https://github.com/steven-bao1/RATIO.