Search papers, labs, and topics across Lattice.
The paper introduces Question-guided Visual Compression with Memory Feedback (QViC-MF), a novel framework for long-term video understanding that addresses the limitations of independently compressing video frames. QViC-MF employs a Question-guided Multimodal Selective Attention (QMSA) mechanism to selectively preserve visual information relevant to a given question from both the current clip and past related frames stored in memory. Experiments demonstrate that QViC-MF significantly outperforms state-of-the-art methods on long-term video understanding tasks, achieving gains of 6.1% on MLVU test, 8.3% on LVBench, 18.3% on VNBench Long, and 3.7% on VideoMME Long.
By selectively attending to question-relevant information across video frames and memory, QViC-MF achieves state-of-the-art results in long-term video understanding, highlighting the importance of feedback-driven perception.
In the context of long-term video understanding with large multimodal models, many frameworks have been proposed. Although transformer-based visual compressors and memory-augmented approaches are often used to process long videos, they usually compress each frame independently and therefore fail to achieve strong performance on tasks that require understanding complete events, such as temporal ordering tasks in MLVU and VNBench. This motivates us to rethink the conventional one-way scheme from perception to memory, and instead establish a feedbackdriven process in which past visual contexts stored in the context memory can benefit ongoing perception. To this end, we propose Question-guided Visual Compression with Memory Feedback (QViC-MF), a framework for long-term video understanding. At its core is a Question-guided Multimodal Selective Attention (QMSA), which learns to preserve visual information related to the given question from both the current clip and the past related frames from the memory. The compressor and memory feedback work iteratively for each clip of the entire video. This simple yet effective design yields large performance gains on longterm video understanding tasks. Extensive experiments show that our method achieves significant improvement over current state-of-the-art methods by 6.1% on MLVU test, 8.3% on LVBench, 18.3% on VNBench Long, and 3.7% on VideoMME Long. The code will be released publicly.
