Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors evaluate whether autonomous vision-language model (VLM) agents can replace human-in-the-loop searchers in complex, interactive video retrieval tasks such as the Video Browser Showdown. By coupling a VLM controller to an existing video search engine's indexing and querying API, the system autonomously analyzes top-ranked candidate frames and executes iterative query reformulations from a single initial intent description. The resulting autonomous agent matches the retrieval performance of historical human expert operators across several standard interactive benchmark settings.
Autonomous VLM agents can rival seasoned human experts at multi-turn video retrieval without prior exposure to the video corpus, requiring only an initial intent description and search engine access.
Searching large video collections is typically an interactive process in which users play two roles. First, they hold the search intent: the underlying goal that determines what content they seek and why. Second, users must operationalize this intent through an iterative search loop. Users translate their intent into queries, browse the retrieved candidates, and refine their queries based on the results. In this paper, we investigate the capabilities of modern Vision Language Models (VLM) and agentic approaches to reach search goals interactively and fully autonomously. Specifically, we study whether a provided initial specification of a search goal might be sufficient to solve traditionally interactive search tasks with an agentic system. Provided that the involved VLMs are not aware of the whole large video dataset in advance, the key challenge lies in the effective combination of an existing interactive video search system and a smart VLM agent controlling the system. While the search system provides indexing and efficient querying, the VLM-based agents analyze top-ranked items and make decisions about next actions. Our results show that modern agents can autonomously operate interactive video retrieval systems to solve many search tasks from an initial intent description, achieving performance competitive with strong historical expert-operated systems in several settings.