Search papers, labs, and topics across Lattice.
This paper develops a theoretical framework for understanding acceptance criteria in speculative decoding, focusing on practical systems that utilize greedy decoding and relaxed acceptance rules. By characterizing rejection regions as lower level sets of the target distribution, the authors derive exact KL divergence metrics and margin-based bounds for various acceptance methods, including strict greedy and tree-based decoding. The evaluation on Qwen3 models reveals that these relaxed criteria significantly expand the certified acceptance region, particularly in scenarios with low distribution margins, enhancing the practical applicability of speculative decoding techniques.
Relaxed and tree-based acceptance criteria can dramatically increase the certified acceptance region in speculative decoding, challenging traditional views on distribution preservation.
Speculative decoding accelerates language model inference by using a fast drafter to propose candidate tokens that are then verified by a larger target model. Existing theory largely studies the stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.
