Search papers, labs, and topics across Lattice.
This study investigates the impact of visual token pruning in vision-language models (VLMs) by analyzing the functional roles of tokens using EmbedLens. The authors reveal that existing pruning methods exhibit distinct biases towards certain token roles, yet these biases do not consistently correlate with improved downstream performance. Notably, their findings indicate that retaining non-alive tokens can enhance performance, challenging the assumption that only semantically strong tokens are beneficial during pruning.
Retaining seemingly redundant tokens can actually enhance model performance, defying traditional assumptions about token importance in visual processing.
Vision-language models (VLMs) process an image as a sequence of visual tokens, which creates a substantial computational bottleneck during inference. Recent visual token pruning methods address this issue by removing seemingly redundant tokens, yet it remains unclear how these pruning decisions relate to the functional roles of visual tokens. In this work, we analyze visual token pruning through the lens of token roles identified by EmbedLens. We first show that representative pruning methods exhibit distinct token-role biases, but these biases do not directly correlate with downstream performance. To better understand this behavior, we refine the token-role assignment procedure and evaluate role-protected pruning variants. Our results show that preserving non-alive tokens can sometimes maintain or improve performance, suggesting that tokens with weak direct semantic alignment may still affect model behavior under pruning. Our code is publicly available at https://github.com/jaykim9870/Not_All_Redundant_Tokens_Are_Alike.
