Search papers, labs, and topics across Lattice.
This paper critiques the conventional practice of decoding from the final layer of large language models (LLMs) by identifying a Guess-Refine-Perturb dynamic across layers, where earlier layers provide initial guesses that are refined by intermediate layers but can be negatively impacted by final-layer perturbations. The authors introduce Confident Decoding, a novel strategy that employs entropy-guided backward search to select the most reliable near-final layer for generating predictions, effectively mitigating alignment-induced noise. Experimental results across various reasoning benchmarks indicate that this approach yields significant improvements in reasoning performance without incurring additional memory costs or substantial latency increases.
Bypassing final-layer perturbations can significantly enhance reasoning capabilities in aligned LLMs, achieving better performance with zero memory overhead.
Autoregressive generation in large language models (LLMs) conventionally decodes from the final layer, assuming that deeper representations yield more reliable next-token predictions. We revisit this assumption by revealing a recurring Guess-Refine-Perturb dynamic: early layers form coarse guesses, intermediate layers refine reasoning-relevant semantics, and final layers can perturb these refined predictions toward generic or alignment-preferred tokens. We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer through entropy-guided conservative backward search. We further provide a theoretical formulation of layer selection as an optimal stopping problem, showing that under bounded projection noise and dominant late-stage alignment perturbation, our search rule filters perturbation while bounding the loss relative to the oracle refinement layer. Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks, including GPQA-Diamond, Omni-MATH, and HLE, with zero memory overhead and less than 2% latency increase. These results suggest dynamically bypassing final-layer perturbations can unlock stronger reasoning behavior from aligned LLMs.
DAMO
Tsinghua AI