Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FORGE, a model-agnostic method that optimizes the selection of relevant frames from long-form videos by inducing a query-conditioned geometry in a pretrained multimodal embedding space. By maximizing the diversity and relevance of selected frames at inference time, FORGE significantly enhances keyframe selection and question answering performance across various multimodal large language models. Experimental results demonstrate an improvement of 11.0-15.3 points in keyframe selection scores and up to 8.7 points in accuracy for question answering tasks, highlighting the effectiveness of aligning the embedding space with query structures.
FORGE doubles keyframe recall and boosts question answering accuracy by aligning frame selection with query-relevant content, transforming long-form video understanding.
Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible. However, the density of relevant content decreases sharply as video sequence length increases, and exposing the model to more irrelevant content measurably reduces its accuracy. In this paper, we address the problem of maximizing query-relevant information in a frame subset selected at inference time, without training. FORGE (Frame Orthogonality in Relevance Geometry) is a model-agnostic method that induces a query-conditioned geometry on a pretrained multimodal embedding space, unifying relevance and diversity into a single objective. In this space, frames that cover independent query-relevant directions are far apart, and selecting the subset of maximum information captures diverse query-relevant content within the budget. Experiments on Video-MME and LongVideoBench at budgets of 16, 32, and 64 frames show that FORGE improves the unified keyframe selection score by 11.0-15.3 points over the strongest training-free baseline and up to doubles keyframe recall (0.415 vs. 0.204 at K=64 on Video-MME). The gains extend to question answering, where accuracy improves in every evaluated setting across eight open-source MLLMs spanning 4B to 32B parameters, by up to 8.7 points over uniform sampling and 5.2 points over the strongest baseline. Our findings suggest that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.