Search papers, labs, and topics across Lattice.
The paper introduces Learning to Draft (LTD), a reinforcement learning-based approach for adaptive speculative decoding that optimizes the throughput of each draft-and-verify cycle. LTD trains two co-adaptive policies to dynamically coordinate the draft and verification phases, unlike existing methods that use static time allocation or proxy metrics. Experiments across five LLMs and four tasks demonstrate that LTD achieves speedups of 2.24x to 4.32x, surpassing the state-of-the-art Eagle3 method by up to 36.4%.
Speculative decoding gets a throughput boost of up to 4.32x by using reinforcement learning to dynamically balance drafting and verification.
Speculative decoding accelerates large language model (LLM) inference by using a small draft model to generate candidate tokens for a larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the-art methods rely on a static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), a novel method that directly optimizes for throughput of each draft-and-verify cycle. We formulate the problem as a reinforcement learning environment and train two co-adaptive policies to dynamically coordinate the draft and verification phases. This encourages the policies to adapt to each other and explicitly maximize decoding efficiency. We conducted extensive evaluations on five diverse LLMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.
Microsoft Research