Search papers, labs, and topics across Lattice.
This paper introduces FOVEA, a novel approach for multimodal speculative decoding that optimizes visual evidence usage by dynamically retrieving a subset of visual inputs based on task demands. By employing a cumulative-mass rule for selection and integrating a lightweight gated residual correction, FOVEA enhances the draft model's performance without the overhead of excessive visual context. Experimental results indicate that FOVEA achieves up to a 2.13x speedup in decoding speed while improving draft acceptance across various vision-language models and benchmarks.
Dynamic visual evidence retrieval can accelerate multimodal decoding by over 2x while enhancing draft acceptance rates.
Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically condition the drafter on a fixed visual interface, such as a predefined visual-token budget or a static compressed representation. However, our controlled visual-budget analysis shows that visual demand varies substantially across tasks and decoding stages, which means more visual input is not always beneficial. Actually, insufficient evidence may weaken visual grounding, while excessive context adds overhead and may disrupt drafting. We propose FOVEA (Focused On-demand Visual Evidence Adaptation), a cache-friendly approach that builds a reusable visual memory and dynamically retrieves a bounded subset for a draft state. A cumulative-mass rule determines both how many and which entries are selected. The selected entries are aggregated into a visual readout and fused with the current draft hidden state through a lightweight gated residual correction. Rather than inserting visual tokens into the autoregressive context, the correction modifies only the representation passed to the language-model head. Experiments across multiple vision-language backbones and multimodal benchmarks show that FOVEA improves draft acceptance and end-to-end decoding speed, achieving up to $2.13\times$ speedup over autoregressive decoding. These results demonstrate that state-conditioned evidence retrieval is an effective alternative to reusing a fixed visual representation throughout multimodal generation.
