Search papers, labs, and topics across Lattice.
This paper introduces LibraSpec, a novel approach to dynamic speculative decoding that optimizes the speculative length based on a marginal-gain-driven criterion, shifting the focus from how many tokens to generate to the value of verifying those tokens. By leveraging drafter confidence scores, LibraSpec iteratively determines the optimal speculative length without requiring additional training. Experimental results demonstrate that LibraSpec achieves significant speedups, with improvements of 0.5 to 1.5 times over existing methods and up to 8.49 times faster than traditional autoregressive decoding.
Speculative decoding can be up to 8.49 times faster than autoregressive methods with LibraSpec's novel marginal-gain-driven optimization.
Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification, with efficiency critically determined by the speculative length selected at each decoding round. Existing dynamic speculation methods select the speculation length by estimating how many tokens will be accepted, which is reasonable for autoregressive drafters that generates tokens sequentially. The recent wave of diffusion-based drafters, however, generates candidate blocks in parallel at substantially lower drafting cost, shifting the key question from how many tokens to generate to how many generated tokens are worth verifying. We therefore reformulate dynamic speculative-length selection as expected-speedup optimization and derive a marginal criterion that extends the speculative sequence only when its acceptance gain outweighs the additional verification cost. Building on this criterion, we develop \textit{LibraSpec}, a training-free and plug-and-play algorithm that iteratively determines the speculative length using drafter confidence scores. Theoretically, we prove that LibraSpec monotonically converges toward the optimal speculative length. Experiments across six target models, three diffusion-based speculative decoding methods, and math, coding, and chat benchmarks show consistent improvements under both greedy and sampling settings, achieving a further $0.5\sim1.5\times$ improvement over baselines and up to $8.49\times$ speedup over autoregressive decoding.
