Search papers, labs, and topics across Lattice.
This paper introduces BlockPilot, an instance-adaptive policy learning approach that optimizes block size selection in diffusion-based speculative decoding, addressing the limitations of fixed inference block sizes. By recognizing that the optimal block size varies across samples and is influenced by the local structure around the training block size, BlockPilot significantly enhances decoding performance. Experimental results show that this method achieves a 4.20x speedup in inference with minimal overhead, demonstrating its effectiveness in improving efficiency in state-of-the-art models like Qwen3-4B.
Optimal block size selection can lead to a 4.20x speedup in diffusion-based speculative decoding, revolutionizing inference efficiency.
Speculative decoding accelerates inference by using a lightweight draft model to generate candidate tokens in parallel, and are then verified by the target model, enabling lossless acceleration. Recently, diffusion-based speculative decoding further improves parallelism by generating multiple tokens per forward pass via block-level diffusion, achieving state-of-the-art (SOTA) performance. However, existing methods adopt a fixed inference block size and assume a uniform optimal decoding strategy across all inputs. In this paper, we show that this assumption is suboptimal, as the optimal block size varies across samples and plays a critical role in speculative decoding performance. Moreover, these values exhibit a clear local structure, concentrating around the training block size, which reduces the problem to a low-dimensional and structured decision space. Based on these insights, we propose BlockPilot, a sample-adaptive policy that predicts the optimal block size from the prefilling representation. Specifically, we formulate block size selection as a lightweight policy learning problem and propose an instance-adaptive decision mechanism that predicts the optimal block size based on the representation of the prefilling stage. The prediction is performed only once after prefilling, allowing for seamless integration. Extensive experiments demonstrate that our method is plug-and-play, introduces minimal overhead, and consistently improves efficiency, achieving an acceptance length of 5.92 and a 4.20times speedup on Qwen3-4B under temperature T=1.
