Search papers, labs, and topics across Lattice.
This paper introduces a training-free token pruning framework called Cross-Layer Spectral Evolution (CLSE) that enhances the efficiency of Multimodal Large Language Models (MLLMs) by addressing the limitations of existing single-layer token pruning methods. By analyzing how token representations evolve across Transformer layers in the frequency domain, CLSE identifies semantically active tokens that should be preserved, thereby reducing visual token redundancy without sacrificing cross-modal reasoning performance. Experimental results show that CLSE significantly reduces computational costs while maintaining or improving model accuracy across various benchmarks in image and video tasks.
Spectral analysis reveals that tokens with dynamic cross-layer evolution are crucial for preserving semantic integrity, leading to more efficient MLLMs.
Reducing visual token redundancy is critical for accelerating Multimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performance. Existing token pruning methods typically rely on single-layer signals, such as attention scores or token similarities, which overlook the cross-layer transformation of visual representations and may exhibit positional bias in multimodal token sequences. To address this limitation, we propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE). Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain. This evolution reflects the transition from high-frequency structural details to low-frequency semantic abstractions. We observe that tokens with stronger spectral redistribution across layers are more likely to be semantically active and should therefore be preserved. By modeling cross-layer token dynamics, CLSE provides a stable importance criterion that mitigates positional bias. Extensive experiments on both image and video benchmarks demonstrate that CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction. Across multiple MLLMs, CLSE reduces FLOPs, KV cache memory, and latency while maintaining competitive or improved performance.
