Search papers, labs, and topics across Lattice.
This paper introduces VIDEOHARNESS-RSI, a method for optimizing context construction in long-video understanding by recursively searching executable context constructors around a frozen vision-language model (VLM). By employing an outer-loop proposer that generates and evaluates candidate harnesses based on prior programs and execution traces, the study reveals significant improvements over both uniform sampling and weaker hand-crafted baselines. Notably, the selected harness demonstrates transferability to additional long-video benchmarks, establishing a new optimization layer for context construction in VLMs.
Recursive search can unlock substantial gains in long-video understanding, outperforming traditional hand-crafted approaches.
Long-video understanding depends critically on how a limited model context is constructed from a much longer video. Existing approaches improve this process through compression, retrieval, memory, and agentic evidence acquisition, but these mechanisms are typically introduced as part of a manually designed inference system or optimized together with other components. This makes it difficult to isolate a simpler question: how much can be gained by improving the executable context-construction program alone? We study this question through VIDEOHARNESS-RSI, a controlled baseline for recursively searching executable context constructors around a frozen vision-language model (VLM). An outer-loop proposer uses prior programs, evaluation outcomes, and execution traces to generate candidate harnesses, which are executed and evaluated end to end before successful variants are retained for further search. This makes long-video understanding a controlled instance of automated harness design: the searchable object is executable program structure, while the answering model and interface remain fixed. Starting from uniform sampling, recursive harness search consistently finds room for improvement and surpasses several weaker hand-crafted baselines. Starting instead from a stronger hand-crafted baseline, the same RSI process yields a further improvement. The selected harness also transfers to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide a reproducible baseline for studying harness discovery and transfer around frozen VLMs.
