Search papers, labs, and topics across Lattice.
This paper introduces FinRAG-QA, a comprehensive benchmark dataset designed to enhance financial question answering by addressing the complexities of banks' financial statements across multiple jurisdictions. The dataset includes 999 curated questions based on 10 standardized indicators, derived from 209 reports of 24 major banks, enabling cross-institutional analysis of lengthy documents. Key results show that a multi-stage retrieval-augmented generation (RAG) pipeline significantly improves answer accuracy and retrieval effectiveness, with a notable increase in NDCG@10 from 0.322 to 0.710 and answer accuracy rising from 44.6% to 79.0%.
A novel benchmark dataset reveals that a multi-stage RAG pipeline can dramatically boost financial question answering accuracy by over 34 percentage points.
The comparative analysis of banks'financial statements poses significant challenges for automated question answering systems due to their complexity, substantial length, technical language, and inhomogeneity of both textual and numerical content across different jurisdictions and institutions. We introduce FinRAG-QA, a novel benchmark dataset for financial question answering, which comprises 999 practitioner-curated questions on 10 standardised indicators, grounded in 209 annual and Pillar 3 reports from 24 major European and U.S. banks spanning 2019-2023. Unlike prior financial QA benchmarks, which centre on U.S. filings and single-institution analysis, FinRAG-QA targets cross-institutional retrieval over documents averaging 198k words, longer than any existing financial QA resource. On this benchmark we evaluate a multi-stage RAG pipeline and isolate the contribution of each component. Contextual chunk enrichment combined with a retrieval-optimised embedding model raises NDCG@10 from 0.322 to 0.710; conditional on the ground truth being retrieved, a reasoning-optimised generator raises answer accuracy from 44.6% to 79.0% (+34.4 percentage points), at roughly 20x the generation latency. We further show that cross-encoder reranking degrades retrieval when the first-stage ranking is already strong, and that a single top-ranked chunk outperforms larger contexts at generation time. Experiments were run in late 2024-early 2025 with the models available at that time.
