Search papers, labs, and topics across Lattice.
The paper introduces VIA-SD, a novel multi-tier framework for speculative decoding that leverages intra-model routing to optimize the verification process of candidate tokens generated by lightweight drafters. By employing a slim-verifier for moderate confidence cases, the method significantly reduces reliance on full model calls, leading to substantial speedups and lower rejection rates across various tasks. The results indicate that VIA-SD achieves 10-20% faster inference compared to existing speculative decoding methods while maintaining compatibility with current training procedures.
Rejecting tokens isn't always the end鈥擵IA-SD's slim-verifier can accurately validate many candidates, slashing inference costs and boosting efficiency.
Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and delivers 10-20% speedups over strong SD baselines, while achieving 2.5-3x acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/
