Search papers, labs, and topics across Lattice.
The paper introduces Spatial Token Fusion (STF) and Multi-Block Token Fusion (MBTF) to reduce the computational cost of Large Multimodal Models (LMMs) by shortening vision token sequences. STF fuses spatially adjacent tokens, while MBTF supplements multi-granularity features for the reduced token sequence to compensate for information loss. Experiments on LLaVA-1.5 show that the proposed method achieves comparable or superior performance on vision-language benchmarks using only 25% of the original vision tokens.
LLaVA-1.5 can achieve similar or better vision-language performance with only 25% of the original vision tokens by spatially fusing redundant tokens.
Large multimodal models (LMMs) suffer significant computational challenges due to the high cost of Large Language Models (LLMs) and the quadratic complexity of processing long vision token sequences. In this paper, we explore the spatial redundancy among vision tokens and shorten the length of vision token sequences for inference acceleration. Specifically, we propose a Spatial Token Fusion (STF) method to learn compact vision tokens for short vision token sequence, where spatial-adjacent tokens are fused into one. Meanwhile, weight-frozen vision encoder can not well adapt to the demand of extensive downstream vision-language tasks. To this end, we further introduce a Multi-Block Token Fusion (MBTF) module to supplement multi-granularity features for the reduced token sequence. Overall, we combine STF and MBTF module to balance token reduction and information preservation, thereby improving inference efficiency without sacrificing multimodal reasoning capabilities. Experimental results demonstrate that our method based on LLaVA-1.5 achieves comparable or even superior performance to the baseline on 8 popular vision-language benchmarks with only $25\%$ vision tokens of baseline. The source code and trained weights are available at https://github.com/visresearch/LLaVA-STF.
