Search papers, labs, and topics across Lattice.
This paper introduces an audio-first triage method for budgeted vision-language captioning of untrimmed egocentric videos, addressing the high costs associated with invoking vision-language models (VLMs) on lengthy video content. By selecting windows based on audio features before any video decoding, the proposed approach significantly reduces the number of VLM calls while maintaining action coverage. The results show a 4.0-10.8 percentage point improvement in action coverage and a reduction of 9-20% in VLM calls on the EPIC-KITCHENS-100 dataset, outperforming existing visual keyframe selectors.
Audio-first triage cuts VLM calls by up to 20% while improving action coverage, revolutionizing the efficiency of processing egocentric videos.
Automatically analyzing hours-long egocentric video is increasingly essential for progress monitoring, quality control, and safety in logistics, construction, and manufacturing. Yet current pipelines that process short, fixed-size windows with a vision-language model (VLM) are prohibitively expensive because cost scales with the number of model calls. To reduce this cost, prior work proposes triage policies to select which windows merit a VLM invocation. However, these policies either sample uniformly or rank windows using visual features, which ironically requires the video decoding that the budget constraints are meant to avoid. We propose audio-first triage: select windows using the lightest modality, scored before any video frame is decoded, so the approach composes naturally with token compression or quantization. The novelty lies in the objective, not the representation: rather than a per-frame sound-event detector, we train the selector to trigger once per action. This objective shift improves action coverage by 4.0-10.8 percentage points across all evaluated call rates, using frozen AudioSet-pretrained features without domain-specific sound-event labels. Using fewer than half of the available calls, the triage cuts 9-20% of VLM calls at matched coverage on EPIC-KITCHENS-100 (EK-100), surpasses uniform sampling through the mid-range on Ego4D over 247 clips, and outperforms two recent visual keyframe selectors. Code, the reference implementation and every results file this manuscript reads are at https://github.com/masjalayer/PreDecoding-AcousticTriage.
