Search papers, labs, and topics across Lattice.
This paper introduces Deferred Visual Ingestion (DVI), a novel framework for visual-dense document question answering that postpones computationally expensive VLM processing until question time. DVI indexes documents using lightweight metadata and full-text search for page localization, then applies VLMs to relevant images conditioned on the user's question. Experiments on industrial engineering drawings demonstrate that DVI achieves comparable accuracy to pre-ingestion methods with zero ingestion VLM cost and significantly improved performance on visually-dependent queries.
Stop wasting VLM tokens upfront: this new approach to visual document QA slashes costs by deferring visual processing until a specific question demands it.
Existing multimodal document question answering methods universally adopt a supply-side ingestion strategy: running a Vision-Language Model (VLM) on every page during indexing to generate comprehensive descriptions, then answering questions through text retrieval. However, this "pre-ingestion" approach is costly (a 113-page engineering drawing package requires approximately 80,000 VLM tokens), end-to-end unreliable (VLM outputs may fail to be correctly retrieved due to format mismatches in the retrieval infrastructure), and irrecoverable once it fails. This paper proposes the Deferred Visual Ingestion (DVI) framework, adopting a demand-side ingestion strategy: the indexing phase performs only lightweight metadata extraction, deferring visual understanding to the moment users pose specific questions. DVI's core principle is "Index for locating, not understanding"--achieving page localization through structured metadata indexes and BM25 full-text search, then sending original images along with specific questions to a VLM for targeted analysis. Experiments on two real industrial engineering drawings (113 pages + 7 pages) demonstrate that DVI achieves comparable overall accuracy at zero ingestion VLM cost (46.7% vs. 48.9%), an effectiveness rate of 50% on visually necessary queries (vs. 0% for pre-ingestion), and 100% page localization (98% search space compression). DVI also supports interactive refinement and progressive caching, transforming the "QA accuracy" problem into a "page localization" problem--once the correct drawing page is found, obtaining the answer becomes a matter of interaction rounds.
