Search papers, labs, and topics across Lattice.
This paper introduces a novel scheme that adapts block verification from large language models (LLMs) to continuous diffusion processes, addressing the challenges of efficiently sampling from residual distributions. By implementing the Free Drafter, a heuristic self-speculative drafter that requires no training, the authors achieve a significant improvement in the acceptance rate of drafts. The proposed method results in up to a 6.3% speedup in inference time compared to existing speculative methods, with minimal overhead.
Block verification boosts the efficiency of diffusion models, yielding a surprising 6.3% speedup in inference without additional training.
Speculative decoding speeds up LLM inference by using a draft model to generate tokens, with an acceptance-rejection scheme that ensures that the output matches the target distribution. Adapting this to continuous diffusions is difficult because speculative sampling requires drawing from a residual distribution. While straightforward in discrete spaces, efficiently sampling this residual in continuous space is non-trivial. Consequently, existing diffusion adaptations either use computationally inefficient sampling techniques or rely on an alternative scheme. In this work, we introduce a novel scheme that efficiently implements the original speculative sampling mechanism for diffusion models. Our approach offers a critical advantage over current methods: it enables us to adapt block verification from LLMs to diffusions -- which provably improves the acceptance rate of drafts. Furthermore, we formalize and analyze the Free Drafter, a heuristic self-speculative drafter for diffusions that requires no training. By enabling block verification, our Free Drafter yields up to a 6.3% speedup over existing speculative methods with no additional training and negligible overhead beyond the existing parallel verification pass.
Google Research