Search papers, labs, and topics across Lattice.
The paper introduces Sparrow, a novel framework to improve speculative decoding in Video Large Language Models (Vid-LLMs) by addressing attention dilution and negative visual gain issues. Sparrow employs visually-aware text-anchored window attention with hidden state reuse to offload visual computation to the target model, and uses intermediate-layer visual state bridging to train the draft model with semantic-rich intermediate states. Experiments demonstrate Sparrow achieves a 2.82x speedup, mitigating performance degradation in long sequences and enabling real-time long video tasks.
Sparrow unlocks 2.8x faster inference for Video LLMs on long videos by cleverly offloading visual computation to the target model using text-anchored attention and semantic-rich intermediate states.
Although speculative decoding is widely used to accelerate Vision-Language Models (VLMs) inference, it faces severe performance collapse when applied to Video Large Language Models (Vid-LLMs). The draft model typically falls into the trap of attention dilution and negative visual gain due to key-value cache explosion and context window mismatches. We observe a visual semantic internalization phenomenon in Vid-LLMs, indicating that critical visual semantics are implicitly encoded into text hidden states during deep-layer interactions, which renders raw visual inputs structurally redundant during deep inference. To address this, we propose the Sparrow framework, which first utilizes visually-aware text-anchored window attention via hidden state reuse to fully offload visual computation to the target model, and leverages intermediate-layer visual state bridging to train the draft model with semantic-rich intermediate states, thereby filtering out low-level visual noise. Additionally, a multi-token prediction strategy is introduced to bridge the training-inference distribution shift. Experiments show that Sparrow achieves an average speedup of 2.82x even with 25k visual tokens, effectively resolving the performance degradation in long sequences and offering a practical solution for real-time long video tasks.
