Search papers, labs, and topics across Lattice.
This paper introduces Oilbird, a training-free speculative decoding method that enhances draft generation by utilizing a re-keyed semantic draft source based on the verifier's hidden state. The authors identify that traditional exact matching fails to capture valuable context in tool-calling scenarios, leading to significant missed opportunities for correct continuations. By integrating this new approach, Oilbird achieves a 24-29% increase in accepted draft length and significantly outperforms existing training-free baselines in decoding speed.
A semantic re-keying strategy in speculative decoding can boost accepted draft lengths by up to 29% while achieving 4.4x faster decoding speeds.
Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most visibly on tool-calling traffic, where a request repeats almost everything but the few values minted for it, and where one rejected token discards the correct continuation behind it. We diagnose the failure position by position across ten benchmarks and find it to be a problem of addressing rather than of coverage: on our densest tool-calling benchmark, about half of what the strongest exact-match drafter misses is present in the pool yet unreachable by exact matching. We therefore propose a second, semantic draft source: the same pool, re-keyed by the hidden state the verifier has already computed at each committed token, together with a merge that lets it ride inside an existing lexical drafter's tree. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.
