Search papers, labs, and topics across Lattice.
This paper introduces the Query- and Content-Aware (QCA) keyframe selection framework, which effectively reduces temporal redundancy in long video understanding by selecting a compact set of semantically relevant frames based on query relevance and content deviation. By partitioning videos into segments and dynamically allocating keyframe budgets, QCA enhances the diversity and relevance of selected frames without requiring additional training. Experimental results show that QCA outperforms existing methods, achieving state-of-the-art performance on benchmarks like LongVideoBench, demonstrating its strong generalization capabilities.
QCA selects the most relevant frames from long videos, achieving superior performance with half the data of previous methods.
Video understanding is often plagued by severe temporal redundancy, where processing dense frame sequences is both semantically inefficient and computationally expensive. This challenge is further amplified when only a small subset of frames is truly relevant to the given query. In this paper, we propose a Query- and Content-Aware (QCA) keyframe selection framework that can select a compact yet information-rich set of frames from long videos. QCA first partitions the video into temporal segments and estimates the information contribution of each segment by jointly modeling query relevance and content deviation, and dynamically allocates keyframe budget to each segment. Within each segment, QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query. Crucially, our method requires no additional training and can be seamlessly integrated into existing Video-LLMs. Extensive experiments across multiple long video understanding benchmarks demonstrate that our proposed approach achieves state-of-the-art performance and has strong generalization ability. For instance, QCA achieves 67.8\% on LongVideoBench using 128 frames, while GPT-4o achieves 66.7\% using 256 frames. Our codes are available in \href{https://github.com/hktk07/QCA}{GitHub}.
