Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To address coupled data and computational bottlenecks in multimodal LLMs, the authors developed SPIDER, a training-free framework combining multi-layer visual token pruning with adaptive sub-layer skipping. The approach exploits the observation that intermediate vision encoder layers retain granular, object-centric details abstracted away by deeper layers, while decoupling attention and FFN contributions across the decoder. On LLaVA-NeXT-7B, SPIDER slashes computational FLOPs by 79% while retaining 96% of the model's baseline multimodal capability.
Standard visual token pruning targets the wrong encoder depth: tapping into intermediate representations while asymmetrically skipping decoder sub-layers cuts MLLM inference FLOPs by 79% without requiring any retraining.
Multimodal Large Language Models face significant efficiency challenges that stem from two distinct yet coupled sources: data redundancy and computational redundancy. While most methods focus on data redundancy by pruning visual tokens from the output of the visual encoder or computing redundancy in LLM decoders using blockwise importance, the finer-grained inter-layer representation shifts and the distribution differences within the layers themselves have not been fully explored. In this work, we comprehensively investigate this dual-level inefficiency. We posit that intermediate layer tokens from vision encoders should be considered for effective visual token pruning, as semantic focus shifts across layers, with middle-layer tokens capturing more detailed object-centric information that deeper layers may abstract away. Furthermore, we reveal the differential contributions of Attention and FFNs across distinct LLM decoder layers. Building upon these discoveries, we propose \textbf{SPIDER}, a training-free framework that integrates multi-layer \underline{\textbf{S}}emantic visual token \underline{\textbf{P}}run\underline{\textbf{I}}ng with an a\underline{\textbf{D}}aptive sub-lay\underline{\textbf{ER}} skipping mechanism. Experimental evaluations demonstrate that SPIDER consistently maintains strong performance across various MLLM architectures and reduction ratios. For instance, on LLaVA-NeXT-7B, SPIDER reduces FLOPs by $79\%$ while maintaining 96$\%$ of the baseline performance.