Search papers, labs, and topics across Lattice.
Tsinghua AIHKUSTThe Hong Kong University of Science and TchnologyUniversity of Sussex 鈭桬qualJul 30, 2026arXiv:2607.28374This paper introduces LedgerMind, a novel framework for multimodal agents that utilizes a Structured Evidence Ledger to ensure provenance-constrained reasoning in visual question answering tasks. By normalizing tool outputs and enforcing strict citation rules, LedgerMind addresses common failure patterns such as unsupported reasoning and entity hallucination, which are often masked by final-answer accuracy metrics. Experimental results demonstrate that LedgerMind not only enhances answer accuracy but also improves the faithfulness of the reasoning trajectories across various multimodal benchmarks.
LedgerMind reveals that grounding multimodal reasoning in a structured evidence ledger can significantly mitigate common pitfalls like entity hallucination and unsupported reasoning.
Multimodal agents for visual question answering increasingly operate as multi-step trajectories that interleave perception, retrieval, and reasoning, yet evaluation still largely reduces to final-answer accuracy. This aggregate signal cannot tell whether a correct answer was reached through grounded evidence, language priors, or accidental error cancellation. We propose to treat a multimodal agent trajectory as a provenance-constrained state machine: tool outputs are normalized into a Structured Evidence Ledger that serves as the trajectory state, downstream reasoning and decision claims may cite only active ledger entries, grounding is checked at the entity and numeric level, and repair is realized as typed state transitions that cannot introduce content without tool-produced provenance. We instantiate this design as LedgerMind (Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger), augmented by a Three-Layer Grounding Protocol, an Adaptive Dual-Path Dispatcher that matches reasoning depth to question complexity, and an Event-Triggered Verification-and-Repair engine with a formal provenance non-amplification guarantee. We use LedgerMind to target four recurring failure patterns that final-answer accuracy tends to obscure: unsupported intermediate reasoning, citation-backed entity hallucination (Phantom Grounding), over-reasoning on simple queries, and repair-time amplification. Experiments across multiple multimodal reasoning benchmarks and backbone MLLMs show that LedgerMind improves both answer accuracy and trajectory-level faithfulness.
