Search papers, labs, and topics across Lattice.
This paper introduces AgentFinVQA, a multi-agent pipeline designed for auditable financial chart question answering that operates on-premise, addressing the need for transparency and trust in regulated environments. By decomposing queries into distinct tasks and recording each step in a traceable Model Evaluation Packet (MEP), the system achieves significant accuracy improvements over existing models, with a +7.68 percentage point gain over a proprietary baseline. The findings highlight the importance of the verifier's output as a confidence signal, revealing that misunderstandings and extraction errors are the primary sources of failure, thus paving the way for future enhancements in financial chart QA systems.
Auditable financial chart QA is now achievable on-premise without sacrificing accuracy, revealing critical insights into model failures and trustworthiness.
Financial chart question answering in regulated settings demands more than accuracy: practitioners must know which answers to trust before acting on them, and many institutions cannot send client data to external model providers. Yet existing chart-QA agents are accuracy-focused and opaque, and most assume proprietary API access; to our knowledge, none combines auditability with on-premise deployability without significant accuracy compromise. We present AgentFinVQA, a multi-agent pipeline that decomposes each query into planning, OCR, legend grounding, visual inspection, and verification, recording every step in a traceable Model Evaluation Packet (MEP) per sample. On FinMME, AgentFinVQA improves $+7.68$ pp over a primary-backbone matched zero-shot baseline with a proprietary backbone (Gemini-3 Flash; 71.24% vs. 63.56%, McNemar $p \approx 1.1 \times 10^{-16}$), and $+4.84$ pp with open-weights Qwen3.6-27B-FP8 served locally. The verifier's verdict also serves as a useful confidence signal (68.2% vs. 55.6% exact accuracy on confirmed vs. revised answers), enabling human-in-the-loop review routing. Error analysis shows that question misunderstanding, legend confusion and extraction error account for nearly two-thirds of failures and are the categories least detected by the verifier, identifying clear directions for future work. Together these results show that auditable, on-premise financial chart QA is practical and that the open-weights system keeps most of the accuracy gains while enabling full data residency. We release our code to support reproducible evaluation.
