Search papers, labs, and topics across Lattice.
This paper addresses the instability of vision token pruning methods under ultra-low token budgets by analyzing the correlation between accuracy degradation and feature distribution shifts. The authors introduce a lightweight distribution consistency metric to quantify these shifts and propose a two-stage pruning framework that includes Anchor-Context Graph Recovery (ACGR) and Text-Aware Token Cluster Selection (TATCS) to mitigate performance loss. Experimental results show that their approach maintains 92.1% of the upper-bound average performance on LLaVA-1.5-7B with just 16 visual tokens, demonstrating enhanced stability in low-token scenarios.
Ultra-low token pruning can achieve 92.1% of peak performance with only 16 visual tokens, thanks to a novel approach that preserves distribution consistency.
Recent vision token pruning methods effectively preserve model performance under moderate token budgets but become unstable under ultra-low token budget. Our analysis shows that as the pruning budget decreases, accuracy degradation is often accompanied by larger feature distribution shifts. Critically, the degree of this distribution shift strongly correlates with performance degradation. To better characterize this phenomenon, we introduce a lightweight distribution consistency metric to estimate the distribution shift between retained and full tokens. Motivated by these observations, we propose a two-stage pruning framework consisting of Anchor-Context Graph Recovery (ACGR) and Text-Aware Token Cluster Selection (TATCS). Specifically, ACGR transfers contextual information before token removal, while TATCS dynamically re-selects representative tokens when severe distribution shift is detected. Extensive experiments demonstrate that our method achieves superior and more stable performance under ultra-low token budget. Notably, it retains 92.1% of the upper-bound average performance on LLaVA-1.5-7B with only 16 visual tokens.
