Search papers, labs, and topics across Lattice.
This paper introduces Weaver, an autoregressive adapter that enhances the efficiency of factorized draft models by constructing proposal trees from the top-K marginals, thereby restoring conditional dependencies among proposed tokens. The authors address the degradation of acceptance rates in speculative decoding as the speculative budget increases, demonstrating that their approach achieves a significant 4.37-fold speedup in decoding time. Additionally, Weaver outperforms the optimized DFlash baseline by 24.7%, showcasing its effectiveness in improving the interactivity of autoregressive language models.
Weaver achieves a 4.37-fold speedup in autoregressive decoding while restoring crucial token dependencies, revolutionizing speculative decoding efficiency.
Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.37-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 24.7%.
