Search papers, labs, and topics across Lattice.
This paper introduces the concept of memory-only answerability to evaluate the efficacy of working memory in long-document visual question answering (VQA) agents, highlighting a critical gap in existing evaluations that overlook the quality of memory used to support answers. The proposed Answerable Working Memory (AWM) framework enhances the GRPO reward mechanism by prioritizing answerable terminal working memory, ensuring that agents retain relevant evidence for grounded answering. Experimental results on the MMLongBench-Doc dataset demonstrate that AWM-GRPO significantly improves final-answer accuracy and reduces the rate of memory-missing correct answers compared to existing methods.
A staggering 42.5% of correct answers in long-document VQA can't be derived from terminal working memory alone, revealing a crucial oversight in current evaluation methods.
Long-document visual question answering increasingly relies on VLM agents that retrieve candidate pages, inspect page images, write findings to working memory, and synthesize answers. Working memory should carry answer-supporting evidence across page inspections for later grounded answering, yet existing evaluation mainly checks final-answer correctness and evidence-page access. This creates a memory-quality blind spot: an agent may reach the right page and answer correctly while leaving behind memory too generic or incomplete to support answering once page context is removed. We introduce \emph{memory-only answerability}, a diagnostic that asks whether a reader can answer from the question and terminal working memory alone. Building on this diagnostic, \emph{Answerable Working Memory} (AWM) treats terminal working memory as an answerable evidence artifact, and AWM-GRPO incorporates this signal into the GRPO reward while preserving final-answer priority. Under GRPO, this reward assigns higher advantages to answer-correct trajectories whose terminal working memory remains answerable. On \textsc{MMLongBench-Doc}, even when gold evidence pages are provided, 42.5\% of correct answers still cannot be answered from terminal working memory alone. AWM-GRPO improves final-answer accuracy over the RAG baseline by 8.1 and 11.9 points on \textsc{MMLongBench-Doc} and \textsc{LongDocURL} and reduces the memory-missing-correct rate by 2.7 points over answer-only GRPO.
Tsinghua AI