Search papers, labs, and topics across Lattice.
The paper introduces GOOSE, a training-free speculative decoding framework that constructs anisotropic speculation trees tailored to the varying quality of token sources. By prioritizing high-acceptance context-matched tokens in a deep chain and diversifying with low-acceptance alternatives, GOOSE optimizes token acceptance rates within a fixed verification budget. Experiments across five LLMs (7B-33B) and five benchmarks demonstrate a 1.9-4.3x lossless speedup, surpassing balanced-tree baselines by 12-33%.
Forget balanced trees: exploiting the quality gap between token sources in speculative decoding unlocks 33% faster inference without sacrificing accuracy.
Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass. Candidates are organized as a tree: deeper trees accept more tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget. Existing training-free methods draft from a single token source and shape their trees without distinguishing candidate quality across origins. We observe that two common training-free token sources - n-gram matches copied from the input context, and statistical predictions from prior forward passes - differ dramatically in acceptance rate (~6x median gap, range 2-18x across five models and five benchmarks). We prove that when such a quality gap exists, the optimal tree is anisotropic (asymmetric): reliable tokens should form a deep chain while unreliable tokens spread as wide branches, breaking through the depth limit of balanced trees. We realize this structure in GOOSE, a training-free framework that builds an adaptive spine tree - a deep chain of high-acceptance context-matched tokens with wide branches of low-acceptance alternatives at each node. We prove that the number of tokens accepted per step is at least as large as that of either source used alone. On five LLMs (7B-33B) and five benchmarks, GOOSE achieves 1.9-4.3x lossless speedup, outperforming balanced-tree baselines by 12-33% under the same budget.
