Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper benchmarks ten retrieval strategies for RAG systems on a financial QA task involving mixed text-and-table documents, evaluating Recall@k, MRR, nDCG, and Number Match. The key finding is that a two-stage pipeline combining hybrid retrieval with neural reranking significantly outperforms single-stage methods, achieving a Recall@5 of 0.816 and MRR@3 of 0.605. Surprisingly, BM25 outperforms state-of-the-art dense retrieval in this financial domain, highlighting the importance of considering domain-specific characteristics.
BM25 still beats dense retrieval for financial documents, so don't blindly assume semantic search is always the best RAG starting point.
Retrieval-Augmented Generation (RAG) systems critically depend on retrieval quality, yet no systematic comparison of modern retrieval methods exists for heterogeneous documents containing both text and tabular data. We benchmark ten retrieval strategies spanning sparse, dense, hybrid fusion, cross-encoder reranking, query expansion, index augmentation, and adaptive retrieval on a challenging financial QA benchmark of 23,088 queries over 7,318 documents with mixed text-and-table content. We evaluate retrieval quality via Recall@k, MRR, and nDCG, and end-to-end generation quality via Number Match, with paired bootstrap significance testing. Our results show that (1) a two-stage pipeline combining hybrid retrieval with neural reranking achieves Recall@5 of 0.816 and MRR@3 of 0.605, outperforming all single-stage methods by a large margin; (2) BM25 outperforms state-of-the-art dense retrieval on financial documents, challenging the common assumption that semantic search universally dominates; and (3) query expansion methods (HyDE, multi-query) and adaptive retrieval provide limited benefit for precise numerical queries, while contextual retrieval yields consistent gains. We provide ablation studies on fusion methods and reranker depth, actionable cost-accuracy recommendations, and release our full benchmark code.