Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce SP$^2$ec, a training-free bandit algorithm for adaptive self-speculative decoding in VLMs that addresses the challenge of diverse input modalities hindering consistent throughput. SP$^2$ec exploits a single-peak throughput pattern observed in existing self-speculative methods by minimizing wall-time regret during generation. Experiments show SP$^2$ec consistently improves throughput by up to 11% over prior self-speculative approaches on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B.
A novel bandit algorithm finds the optimal self-speculative decoding configuration for VLMs in real-time, boosting inference speed by up to 11% without retraining.
Speculative Decoding (SD) has demonstrated its efficacy in accelerating large language model inference, but for Vision-Language Models (VLMs), the diverse input modalities make it difficult for a small trained drafter to achieve consistently high throughput. Self-speculative decoding reuses parts of the target model as a drafter which retains the multi-modal capability of the VLMs, yet its efficiency depends on choosing a suitable draft configuration for each prompt. We propose Single-Peak SPeculative decoding (SP$^2$ec), a training-free bandit algorithm that adapts the draft configuration during generation. SP$^2$ec exploits the single-peak throughput pattern observed in existing self-speculative methods, including SWIFT, KnapSpec, SpecVLM, and our simple cosSim. Our algorithmic design explicitly minimizes a novel notion of wall-time regret for each given query, corresponding to the empirical wall-time latency. Theoretical guarantees on the wall-time regret show that SP$^2$ec can exploit the optimal draft configuration, with negligible exploration cost. Extensive experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B across visual and textual tasks show consistent improvements of up to 11\% over the original self-speculative methods, demonstrating the wide applicability of the proposed SP$^2$ec algorithm.
NUS