Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To challenge the necessity of increasingly complex token pruning and selection heuristics in Video LLMs, the authors evaluate SimpleCluster, a training-free baseline based on position-aware cross-frame feature clustering and mean pooling. Evaluated across three Video LLMs and four video understanding benchmarks, this minimalist approach matches or outperforms existing state-of-the-art compression methods, showing remarkable stability even at an extreme 1% token retention budget. Mechanistic analysis reveals that Video LLM performance under compression is driven primarily by preserving the global coverage and local fidelity of the visual feature manifold rather than heuristic token selection.
Training-free feature clustering beats sophisticated token-pruning heuristics in Video LLMs, maintaining competitive downstream accuracy even when discarding 99% of visual tokens.
Video Large Language Models (Video LLMs) have achieved remarkable progress in video understanding, but their inference efficiency is constrained by the large number of visual tokens produced by long videos. Recent video token compression methods increasingly introduce sophisticated strategies for token selection, pruning, and merging. This raises a fundamental question: how much of compression performance can be obtained by simply preserving the structure encoded in the visual representations? We investigate this question with SimpleCluster, a simple and training-free baseline that performs position-aware cross-frame clustering in the visual feature space and represents each cluster using the mean of its original visual features. Extensive experiments across four video understanding benchmarks and three representative Video LLMs show that SimpleCluster achieves competitive or superior performance over recent compression methods across a wide range of token retention ratios, with particularly strong robustness under extremely low retention rates (e.g., 1%). To understand this behavior, we analyze the feature space preserved by different compression methods in terms of local approximation fidelity and global coverage. The results show that stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially its global coverage. These findings highlight feature-space preservation as an important consideration for video token compression under highly constrained token budgets. Our code is available at https://github.com/xiaozhang79/SimpleCluster.