Search papers, labs, and topics across Lattice.
This paper introduces SparseSpec-L, a training-free self-speculative decoding framework designed to optimize long-context inference in large language models by addressing the memory-bandwidth bottleneck. The authors conduct a unified efficiency analysis revealing that extending speculation horizons can be counterproductive when the acceptance probability is low relative to drafting costs. Experimental results demonstrate that SparseSpec-L achieves significant end-to-end acceleration over traditional autoregressive decoding methods while maintaining the integrity of the target model's output distribution.
Extending speculation horizons can backfire, but SparseSpec-L smartly navigates this trade-off to achieve substantial speedups in long-context inference without sacrificing output quality.
Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.
