Search papers, labs, and topics across Lattice.
This paper investigates the failure of visual token pruning methods in Multimodal Large Language Models (MLLMs) on complex visual reasoning tasks, attributing the issue to Relevant Visual Information Shift (RVIS) during decoding. To address this, they introduce Decoding-stage Shift-aware Token Pruning (DSTP), a training-free framework that adapts existing pruning methods to align visual tokens with shifting reasoning requirements. Experiments show DSTP significantly improves performance on complex reasoning tasks and maintains or improves performance on visual understanding benchmarks across various MLLM architectures.
Visual token pruning, effective for simple tasks, breaks down in complex visual reasoning due to shifting information needs during decoding, but a simple add-on fixes it.
Recently, visual token pruning has been studied to handle the vast number of visual tokens in Multimodal Large Language Models. However, we observe that while existing pruning methods perform reliably on simple visual understanding, they struggle to effectively generalize to complex visual reasoning tasks, a critical gap underexplored in previous studies. Through a systematic analysis, we identify Relevant Visual Information Shift (RVIS) during decoding as the primary failure driver. To address this, we propose Decoding-stage Shift-aware Token Pruning (DSTP), a training-free add-on framework that enables existing pruning methods to align visual tokens with shifting reasoning requirements during the decoding stage. Extensive experiments demonstrate that DSTP significantly mitigates performance degradation of pruning methods in complex reasoning tasks, while consistently yielding performance gains even across visual understanding benchmarks. Furthermore, DSTP demonstrates effectiveness across diverse state-of-the-art architectures, highlighting its generalizability and efficiency with minimal computational overhead.
