Search papers, labs, and topics across Lattice.
This paper introduces CoFiE, a Coarse-to-Fine Evidence Selection framework designed to enhance the efficiency of Vision Language Models (VLLMs) in streaming video understanding by decoupling evidence selection into a coarse filtering stage and a fine refinement stage. By employing Novelty-Guided Frame Filtering and Query-Specific Evidence Refinement, CoFiE significantly reduces redundancy before frame encoding while ensuring relevance to user queries. Experimental results demonstrate that CoFiE achieves a state-of-the-art accuracy-efficiency trade-off, improving end-to-end inference latency by up to 2.54 times while outperforming existing multimodal models on key benchmarks.
CoFiE achieves a remarkable 2.54 times improvement in end-to-end inference latency while maintaining high accuracy in streaming video understanding.
Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints. Existing methods improve efficiency through token pruning and memory-bank schemes, but mainly reduce visual tokens after visual encoding. Consequently, downstream token pruning alone cannot substantially reduce end-to-end latency because the expensive frame encoding cost has already been incurred. We propose CoFiE, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill. CoFiE introduces Novelty-Guided Frame Filtering to retain visually distinctive candidate frames and Query-Specific Evidence Refinement to select the frames most relevant to the user query. This design removes substantial redundancy before frame encoding while preserving query-specific refinement once semantic information becomes available. Experiments show that CoFiE establishes a new state-of-the-art accuracy-efficiency trade-off across multiple video understanding benchmarks, reaching 78.86% accuracy on StreamingBench and 68.72% on OvO-Bench, with improvements of up to 3.15% over prior methods. Even with up to 80% evidence-frame filtering, CoFiE outperforms strong open-source multimodal models while improving end-to-end inference latency by up to 2.54 times.
