Search papers, labs, and topics across Lattice.
This paper tackles the trade-off between vocabulary size and latency in draft models for speculative decoding by proposing vocabulary trimming. They formulate draft vocabulary selection as a constrained optimization problem, balancing token coverage (measured on training data) and draft latency (estimated via architecture-aware FLOPs). Using a Tree-structured Parzen Estimator, they efficiently explore the coverage-latency Pareto frontier, achieving up to 16% latency reduction and 20% throughput improvement on domain-specific tasks by reducing draft vocabularies by up to 97%.
Shrinking draft model vocabularies by up to 97% can significantly boost speculative decoding throughput, especially for domain-specific tasks.
Speculative decoding accelerates inference for Large Language Models by using a lightweight draft model to propose candidate tokens that are verified in parallel by a larger target model. Prior work shows that the draft model often dominates speculative decoding latency, since it generates tokens sequentially and incurs high cost from its language modeling head as vocabulary size grows. This exposes a fundamental trade-off in draft model design: larger vocabularies improve token coverage and agreement with the target model, but incur higher draft latency, while smaller vocabularies reduce latency at the risk of missing tokens required for accurate draft generation. We address this trade-off through vocabulary trimming for draft models, motivated by the observation that domain-specific workloads use only a small fraction of the full vocabulary. We cast draft vocabulary selection as a constrained optimization problem that balances token coverage and draft latency. Coverage is computed over assistant responses in the training data, while latency is estimated using architecture-aware FLOPs that capture the cost of the language modeling head as a function of vocabulary size. We optimize a utility function with a Tree-structured Parzen Estimator to efficiently explore the coverage-latency Pareto frontier under a minimum coverage constraint. Experiments show improved speculative decoding throughput while reducing draft vocabularies by up to 97% with high coverage. On domain-specific tasks, we achieve up to 16% latency reduction and 20% throughput improvement, and up to 6.7% throughput gains on diverse out-of-distribution tasks.
