Search papers, labs, and topics across Lattice.
This paper introduces Progressive Tree Drafting (PTD), a novel approach to speculative decoding that enhances the inference speed of Large Language Models (LLMs) by leveraging a structured parallel drafting strategy. By integrating a progressive tree structure with a stepwise pruning mechanism, PTD allows the model to explore multiple semantic paths simultaneously, significantly improving draft diversity and coherence without incurring additional training costs. Experimental results show that PTD can achieve up to a 2x speedup in decoding across various benchmarks, highlighting its effectiveness in overcoming memory-bound bottlenecks in LLM inference.
Achieving up to 2x faster decoding in LLMs without additional training, Progressive Tree Drafting redefines how we approach speculative decoding.
Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks. However, traditional speculative decoding typically relies on auxiliary draft modules, incurring significant training and communication overhead. Although recent methods attempt to generate drafts within the target model itself, they often fail to fully exploit its latent parallel capacity due to a lack of structural coordination. In this paper, we propose \textbf{Progressive Tree Drafting (PTD)}, which employs a structured, guided parallel drafting strategy to harness the model's parallel potential. By coupling a progressive tree structure with a stepwise pruning mechanism, PTD actively guides the LLM to explore multiple semantic paths in a single forward pass, ensuring both draft diversity and coherence. Experiments demonstrate that PTD achieves up to $2\times$ decoding speedup across various benchmarks while remaining training-free and model-agnostic. Our code is available at: https://github.com/MINE-USTC/PTD.
