Search papers, labs, and topics across Lattice.
The paper introduces InfoShot, a novel shot-aware frame sampler for efficient long-video understanding with VLMs. InfoShot partitions videos into semantically consistent shots and selects two keyframes per shot: one representing the main content and another capturing unusual within-shot changes, guided by an information-theoretic objective. Evaluations on a new synthetic benchmark (SynFlash) and existing datasets demonstrate that InfoShot improves anomaly detection and Video-QA accuracy under frame number constraints, while maintaining performance on standard video understanding tasks.
Grabbing two keyframes per shot – one for the gist, one for the glitch – lets you compress videos for VLMs without missing critical anomalies.
Video frame sampling is essential for efficient long-video understanding with Vision-Language Models (VLMs), since dense inputs are costly and often exceed context limits. Yet when only a small number of frames can be retained, existing samplers often fail to balance broad video coverage with brief but critical events, which can lead to unreliable downstream predictions. To address this issue, we present InfoShot, a task-agnostic, shot-aware frame sampler for long-video understanding. InfoShot first partitions a video into semantically consistent shots, and then selects two complementary keyframes from each shot: one to represent the main content and one to capture unusual within-shot changes. This design is guided by an information-theoretic objective that encourages the sampled set to retain high information about both shot structure and sparse within-shot deviations. In this way, it improves the chance of preserving both overall video context and short decision-critical moments without requiring any retraining. To better evaluate such short-lived events, we further introduce SynFlash, a synthetic benchmark with controllable sub-second anomaly patterns and frame-level ground truth, and we also evaluate InfoShot on existing anomaly datasets and general video understanding tasks. Experiments show that InfoShot improves anomaly hit rate and downstream Video-QA accuracy under frame number constraints, while matching or outperforming strong baselines on standard video understanding benchmarks.
