Search papers, labs, and topics across Lattice.
TAP-RAG introduces a task-aware policy-controlled framework for long-document multimodal question answering, addressing the limitations of existing systems that apply a query-agnostic evidence-use strategy. The framework consists of a main controller, the Task-Aware Policy Controller (TAPC), which predicts task priors and generates executable policies, along with two policy-guided evidence executors that enhance evidence retrieval based on query-specific needs. TAP-RAG significantly outperforms existing multimodal retrieval-augmented generation systems, achieving accuracy improvements of +9.1 points and +4.5 points on benchmark datasets DocBench and MMLongBench-Doc, respectively.
TAP-RAG achieves a remarkable +9.1 point accuracy boost over traditional multimodal RAG systems by tailoring evidence retrieval to specific query tasks.
Long-document multimodal question answering requires more than retrieving relevant chunks from a large document. Different queries require different evidence behavior. Existing multimodal RAG systems improve evidence access through text chunks, page images, graph links, or heterogeneous document elements, but they often apply a largely query-agnostic evidence-use strategy. We present TAP-RAG, a task-aware policy-controlled RAG framework for long-document multimodal QA. TAP-RAG contains a main controller, the Task-Aware Policy Controller (TAPC), and two policy-guided evidence executors: Task-Aware Query-Guided Flow Diffusion (TA-QFD) and Task-Aware Visual Enhancement (TAVE). For each query, TAPC predicts the task prior, estimates visual/local/global evidence signals, and produces an executable policy. TA-QFD then expands textual and structural evidence over the multimodal document graph, while TAVE selectively inspects page images when visual or layout evidence is needed. A guarded synthesis stage fuses text, visual, and structural evidence and abstains when support is insufficient. On DocBench and MMLongBench-Doc, TAP-RAG achieves the best overall accuracy among the compared systems, improving over a matched multimodal-RAG baseline by +9.1 points (61.1 to 70.2) and +4.5 points (42.2 to 46.7), respectively.
Tsinghua AI