Search papers, labs, and topics across Lattice.
This paper introduces AdaFlash, an innovative framework that addresses the high variance issues associated with diffusion drafters in speculative decoding by employing an on-policy distillation algorithm and an adaptive length head. The framework stabilizes convergence and reduces both domain-level and token-level variances, leading to more consistent draft quality and improved efficiency. Experimental results show that AdaFlash achieves up to 66% higher throughput compared to existing methods, particularly excelling in high-concurrency environments.
High variance in draft quality is tackled head-on, leading to a remarkable 66% increase in throughput for speculative decoding in large language models.
Speculative decoding, in which a lightweight draft model first generates a draft sequence that is then verified in parallel by the target model, has become a prevalent paradigm for accelerating large language model inference. Recent work such as DFlash further boosts drafting efficiency by leveraging diffusion drafters, whose parallel denoising mechanism enables draft generation in a single forward pass. In this work, we uncover a central pitfall of diffusion drafters: bidirectional attention is a double-edged sword. On one hand, it endows the model with parallel generation and global contextual modeling capabilities; on the other hand, this inherent global dependency introduces high variance at both the domain-level and the token-level: acceptance rates fluctuate substantially across different domains, and draft token quality also varies heterogeneously at different token positions. To tackle this issue, we propose AdaFlash framework, comprising two components: (i) an on-policy distillation (OPD) algorithm with reverse-KL divergence tailored for diffusion drafters, bringing stable convergence and effectively reducing domain-level variance; and (ii) an adaptive length head that dynamically adjusts the candidate sequence length on the fly, substantially lowering the verification cost of the target model and handling token-level variance. Experiments demonstrate that AdaFlash consistently improves speedup rate during deployment, with especially significant gains in high-concurrency scenarios, achieving up to approximately 66% higher throughput than previous state-of-the-art methods.
