Search papers, labs, and topics across Lattice.
This paper introduces ResiSpec, a novel framework designed to enhance the efficiency of multi-candidate speculative decoding in Large Language Models (LLMs) by addressing the issue of Residual Drift, which undermines candidate effectiveness during token validation. By reforming the proposal distribution during the verification phase, ResiSpec ensures that the residual target mass remains aligned with the high-confidence regions of the draft model, thus preventing candidate obsolescence. The approach achieves a significant performance improvement, yielding up to a 1.92脳 speedup compared to existing multi-candidate methods while maintaining output accuracy.
ResiSpec redefines speculative decoding efficiency, achieving nearly double the speed of current methods without sacrificing output quality.
The efficiency of Large Language Model (LLM) serving is fundamentally limited by the sequential nature of autoregressive decoding. Speculative Decoding (SD) mitigates this by using a lightweight draft model to speculate future tokens, which are then validated by the LLM in a single parallel forward pass. To further boost efficiency, multi-candidate schemes propose diverse candidate sets to increase the likelihood of token acceptance. However, we show that these schemes are bottlenecked by Residual Drift: a phenomenon where the rejection of initial candidates causes the residual target distribution to diverge from the draft model's predictions. This shift renders subsequent candidates ineffective and forces the system into expensive resampling. To resolve this, we propose ResiSpec, a framework that strategically reforms the proposal distribution during verification to anchor the residual target mass within the draft model's high-confidence regions. By mathematically re-aligning the verification process without compromising output exactness, ResiSpec prevents candidate obsolescence and achieves up to 1.92$\times$ speedup over state-of-the-art multi-candidate methods. Code is available at https://github.com/Czzzk/Resispec.
