Search papers, labs, and topics across Lattice.
This paper introduces ReSET, a novel temperature-scaling method that enhances reasoning accuracy in large reasoning models (LRMs) under NVFP4 quantization by adapting decoding temperature based on token-level and step-level entropy. The authors identify that quantization leads to increased sampling errors at low-entropy tokens and over-concentration on a limited set of tokens during high-uncertainty reasoning steps. By implementing ReSET alongside a specialized CUDA-core small-$M$ NVFP4 kernel, the approach achieves up to a 2-point accuracy improvement and a 2.5脳 speedup in latency-critical autoregressive decoding compared to existing methods.
ReSET boosts reasoning accuracy in large models while slashing inference latency, achieving a remarkable 2.5脳 speedup in critical decoding tasks.
Large reasoning models (LRMs) improve complex problem-solving by generating long intermediate reasoning traces, but this substantially increases inference costs. NVFP4 inference offers a promising approach to reduce both computational and memory costs through hardware-supported low-precision execution. However, directly applying NVFP4 to LRMs introduces two practical limitations: reasoning accuracy degrades under quantization, and existing NVFP4 kernels do not fully realize latency benefits in small-batch autoregressive decoding. In this work, we analyze the effect of NVFP4 quantization on token-level uncertainty during reasoning. We show that quantization increases incorrect sampling at low-entropy symbolic tokens, while causing over-concentration on a small set of tokens in high-uncertainty reasoning steps. Based on this observation, we propose \textbf{ReSET}, a reasoning-step entropy-based temperature-scaling method that estimates step-level uncertainty online and adapts the decoding temperature using both token-level and step-level entropy signals. To address the latency gap, we further design a CUDA-core small-$M$ NVFP4 kernel for latency-critical autoregressive decoding. Across reasoning benchmarks and model scales, ReSET improves NVFP4 reasoning accuracy by up to $\sim\!$2 points over the NVFP4 baseline. Our CUDA-core small-$M$ kernel further improves latency-critical decoding, delivering up to $2.5\!\times$ kernel-level speedup over NVFP4 vLLM and approximately $2\!\times$ end-to-end decoding speedup over BF16. Code is available at https://github.com/aiha-lab/ReSET.
