Search papers, labs, and topics across Lattice.
This paper introduces an operator-level visual-token skipping framework for multimodal large language models (MLLMs) that selectively bypasses redundant computations while preserving the full visual-token sequence. By analyzing visual-token updates from an answer-observable perspective, the authors identify that late visual-token updates often contribute little to answer-token representations, allowing for significant reductions in computational load. The proposed method achieves a 33.7% reduction in TFLOPs on the Qwen3-VL model while maintaining 99.5% of the original model's performance across various visual question answering benchmarks.
Late visual-token updates can be safely ignored, leading to a 33.7% reduction in computational load without sacrificing performance.
Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answer-silent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing \textbf{33.7\%} TFLOPs on Qwen3-VL while retaining \textbf{99.5\%} of the vanilla model performance.
Tsinghua AI