Search papers, labs, and topics across Lattice.
This paper introduces Matryoshka Evidence-to-Context (MEC) Frame Selection, a novel approach to long-video understanding that optimizes frame selection for varying budget constraints without the need for retraining. By formulating frame selection as a Matryoshka ranking problem, the method constructs a single priority sequence that balances evidence concentration and temporal context across different frame budgets. The results demonstrate that MEC outperforms uniform sampling by 3.77 percentage points in accuracy while significantly reducing selection latency by up to 51.19% across multiple benchmarks.
A single ranking can adapt to any frame budget, improving accuracy and reducing latency without retraining the model.
Frame selection is essential for applying Large Multimodal Models (LMMs) to long videos due to severe frame redundancy and limited context windows. Since the appropriate frame budget varies with the downstream LMM, reasoning demands, and latency constraints, a practical selector should serve multiple budgets. However, existing methods typically optimize an isolated frame subset for each predefined budget: when the budget changes, previously selected evidence may be replaced rather than progressively augmented. Ranking frames by a fixed score would allow prefix reuse across budgets, but it ignores the distinct roles of different ranking positions. In this paper, we formulate long-video frame selection as a Matryoshka ranking problem: constructing a single priority sequence whose small prefixes concentrate query-conditioned evidence, while progressively larger prefixes preserve this evidence and add broader temporal context. Efficiently constructing such a ranking is itself challenging, as densely sampling long videos and evaluating frame-query relevance incurs substantial overhead. We therefore introduce Matryoshka Evidence-to-Context (MEC) Frame Selection, a training-free framework that builds a reusable sparse video index, discovers candidates through sparse probing and local zooming, and greedily constructs a position-adaptive ranking: early positions emphasize evidence; later positions progressively favor temporal coverage while preserving visual diversity. A single ranking can thus be truncated to any target budget without rerunning the selector. Across four benchmarks and six frame budgets, MEC improves average accuracy over uniform sampling by 3.77 percentage points, matches strong state-of-the-art selectors, and reduces end-to-end selection latency by 47.37-51.19%.
Tsinghua AI