Search papers, labs, and topics across Lattice.
This paper introduces VisDocAgentBench, a benchmark designed to evaluate the effectiveness of static versus agentic retrieval methods for visually rich document retrieval, which incorporates language, layout, and visual elements. The study reveals that while a strong visual retriever achieves high recall on direct queries, it struggles significantly on more complex evidence structures, highlighting the limitations of traditional query-target matching. By employing agents, the research demonstrates improved retrieval performance, particularly when leveraging visual information, thus underscoring the importance of iterative search and contextual support in document ranking.
Agents can significantly enhance retrieval performance in visually rich document environments, achieving a 67.50% Recall@1 compared to just 37.50% for OCR-text methods.
Visually rich documents encode relevance through language, layout, structured visual elements, and corpus context, yet retrieval is typically evaluated by one-shot query--page matching. Agentic-search benchmarks usually score downstream question answering or report generation, leaving document ranking under iterative evidence acquisition underexplored. We introduce VisDocAgentBench, a closed-corpus benchmark comparing static and agentic retrieval under a shared ranked-output contract. It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures. Relation-preserving construction yields semantic, relational, and visual queries, followed by full-document review and hard-negative validation. A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context. Agents recover much of this loss, but planner choice and retrieval representation remain decisive. Every planner performs better with visual retrieval, whose best R@1 reaches 67.50% versus 37.50% for OCR-text. Ablations identify iterative search and page inspection as consequential capabilities, and providing the complete support context improves ranking on both routes. Trace analysis localizes the remaining losses to target discovery, candidate examination, and evidence-role integration. These findings motivate retrieval agents that combine modality-preserving discovery with evidence-directed verification.
