Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel approach for query-driven multimodal information extraction from long documents, addressing the limitations of existing methods that primarily focus on textual answers. The authors present ITJoint, a benchmark dataset specifically designed for this task, which includes 2,455 pages of domain-specific documents and evaluates both textual attribute extraction and image localization. Their proposed Q2IT framework, which employs a multi-agent collaborative strategy, demonstrates significant performance improvements over standalone Vision-Language Models, highlighting the complexity of integrating image and text data in response to specific queries.
Query-driven extraction reveals that existing Vision-Language Models falter in multimodal tasks, but a collaborative framework can bridge the performance gap.
In domain-specific multimodal long documents, images and text jointly convey complex knowledge that cannot be fully captured by plain text alone. However, existing paradigms like DocVQA primarily focus on generating textual answers or localizing evidence regions, rather than outputting query-specific textual attribute values and corresponding images. To address this gap, we propose query-driven image-text joint extraction from long documents, requiring models to output query-requested textual attribute values and corresponding image bounding boxes. Based on challenges related to both user intent and document content, we designed a two-level taxonomy that operates at the query and instance levels. Further, we construct ITJoint, the first high-quality, manually annotated benchmark for this new task, comprising 2,455 pages of domain-specific documents with numerous non-decorative images, 316 queries, and 910 answer instances. Finally, we evaluate representative standalone Vision-Language Models from different providers and further design Q2IT, a multi-agent collaborative framework consisting of three progressively collaborating agents for evidence collection, page selection, and target-image localization. Using a joint evaluation approach that assesses both text extraction and image localization, our experiments show that standalone VLMs struggle with this task, while Q2IT significantly improves performance on ITJoint, although a substantial gap remains toward perfect results.