Search papers, labs, and topics across Lattice.
The paper introduces DeepImageSearch, an agentic paradigm for context-aware image retrieval in visual histories, addressing the limitations of existing multimodal retrieval systems that neglect temporal dependencies. They construct DISBench, a challenging benchmark built on interconnected visual data, and propose a human-model collaborative pipeline using vision-language models to generate context-dependent queries. Experiments demonstrate that DISBench poses significant challenges to state-of-the-art models, emphasizing the need for agentic reasoning in retrieval systems.
Current multimodal retrieval systems fall flat when faced with realistic visual streams where context is distributed across time, motivating a new agentic paradigm for context-aware image retrieval.
Existing multimodal retrieval systems excel at semantic matching but implicitly assume that query-image relevance can be measured in isolation. This paradigm overlooks the rich dependencies inherent in realistic visual streams, where information is distributed across temporal sequences rather than confined to single snapshots. To bridge this gap, we introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task. Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues. We construct DISBench, a challenging benchmark built on interconnected visual data. To address the scalability challenge of creating context-dependent queries, we propose a human-model collaborative pipeline that employs vision-language models to mine latent spatiotemporal associations, effectively offloading intensive context discovery before human verification. Furthermore, we build a robust baseline using a modular agent framework equipped with fine-grained tools and a dual-memory system for long-horizon navigation. Extensive experiments demonstrate that DISBench poses significant challenges to state-of-the-art models, highlighting the necessity of incorporating agentic reasoning into next-generation retrieval systems.
Mila