Search papers, labs, and topics across Lattice.
This paper introduces VIA-SD, a multi-tier framework for speculative decoding that optimizes inference costs in large language models (LLMs) by employing a slim-verifier for moderate verification tasks. By utilizing intra-model routing, VIA-SD allows for a hierarchical processing of draft tokens, which leads to significant reductions in rejection rates and inference times compared to traditional draft-verify methods. The results demonstrate that VIA-SD achieves 10-20% speedups over strong baselines while maintaining compatibility with existing speculative decoding frameworks.
Slim-verifiers can accurately validate rejected tokens, leading to a 10-20% speedup in LLM inference without compromising on quality.
Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and delivers 10-20% speedups over strong SD baselines, while achieving 2.5-3x acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/
