Search papers, labs, and topics across Lattice.
This paper explores training-free relaxed speculative decoding techniques, which accelerate sampling from autoregressive language models by allowing for a trade-off between speed and capability. The authors unify existing methods within a shared framework and benchmark their performance, revealing that while relaxed approaches can enhance speed, they often necessitate rigorous capability evaluations and depend on high-quality drafting models. Key findings indicate that many relaxed methods may not be suitable for lightweight multi-token prediction tasks due to their reliance on robust language models.
Relaxed speculative decoding can significantly boost sampling speed, but it comes with hidden costs in capability evaluation and model quality.
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.
