Search papers, labs, and topics across Lattice.
This paper introduces GMM-EVA, a novel framework that utilizes Gaussian Mixture Models to enhance long video understanding by addressing the inefficiencies of uniform keyframe sampling. By strategically allocating visual resources to preserve high-resolution keyframes for key events while using lower-resolution frames for context, GMM-EVA reduces redundancy and computational costs significantly. Experimental results show that GMM-EVA not only outperforms traditional uniform sampling but also achieves comparable performance to existing methods while halving the visual token budget.
GMM-EVA slashes visual token budgets in long video understanding by intelligently prioritizing keyframes based on event-level structures.
Large Vision-Language Models (LVLMs) face significant challenges in long video understanding due to the excessive computational cost and information loss associated with uniform sampling. Existing keyframe selection methods often treat video frames as atomic entities and allocate visual budgets equally, thereby overlooking high-level semantic structures and introducing substantial redundancy. To address these limitations, we propose GMM-EVA (Gaussian Mixture Modeling for Event-Aware Visual Allocation), which leverages Gaussian Mixture Models to model event-level structure from discrete frame-wise observations. A differentiated allocation strategy is then applied to preserve one primary high-resolution keyframe per event for high-fidelity detail, while utilizing lower-resolution secondary keyframes to maintain temporal context and optimize token budgets. GMM-EVA is a training-free, plug-and-play framework that generalizes robustly across various relevance measures and downstream LVLMs. Extensive experiments on multiple long video benchmarks demonstrate that our method significantly outperforms uniform sampling. Notably, GMM-EVA achieves comparable performance to baseline selection methods while utilizing only approximately half of the visual token budget, highlighting its superior efficiency and effectiveness.
