Search papers, labs, and topics across Lattice.
This paper introduces Invoice Haystack, a benchmark designed to evaluate document retrieval and visual question answering in scenarios with high visual homogeneity, addressing the issue of embedding collapse seen in existing benchmarks. By utilizing a dataset of 1,500 anonymized invoice images and 200 question-answer pairs, the authors demonstrate that traditional benchmarks fail to reflect the challenges posed by enterprise document repositories. The proposed VL-RAG framework, which combines text and visual embeddings, achieves a Recall@1 of 60.0% on Invoice Haystack-500, significantly outperforming previous state-of-the-art methods by up to 13.5 percentage points.
Existing benchmarks fail to capture the retrieval challenges posed by visually homogeneous documents, but Invoice Haystack reveals the stark performance drop of current models in this critical area.
Vision Language Models have achieved near-human performance on single-document Visual Question Answering, yet their effectiveness degrades significantly when retrieving information from large collections of visually homogeneous documents. Existing multi-document benchmarks aggregate diverse document types, creating artificial separation in embedding space that does not reflect enterprise document repositories where thousands of records share identical visual templates. We identify this as embedding collapse and introduce Invoice Haystack, a benchmark with 1,500 anonymized invoice images paired with 200 discriminative question-answer pairs, specifically designed to stress-test retrieval under strong visual homogeneity. Invoice Haystack exhibits a mean pairwise cosine similarity of 0.73, compared to 0.38 (DocHaystack) and 0.31 (InfoHaystack) in existing benchmarks, posing a fundamentally more challenging retrieval problem. Addressing the identified challenge, we propose VL-RAG, a hybrid retrieval-augmented generation framework that jointly leverages text and visual embeddings to harness the complementary strengths of both modalities, followed by a VLM-based verification filter for precise document identification. VL-RAG achieves 60.0\% Recall@1 on Invoice Haystack-500, outperforming existing state-of-the-art method by up to an absolute 13.5 percentage points. It further improves retrieval considerably on DocHaystack-1000 (77.1\% vs.\ 75.2\%) and InfoHaystack-1000 (84.5\% vs.\ 80.0\%), establishing the proposed dual-stream fusion as a consistently superior retrieval strategy across both homogeneous and heterogeneous document collections.
