Search papers, labs, and topics across Lattice.
This paper introduces ConceptFormer, a novel framework that learns adaptive latent concepts for enhancing query-document alignment in visual document retrieval. By modeling query-relevant evidence as continuous, query-conditioned latent concepts, the approach effectively bridges the gap between localized visual evidence and semantic relevance without relying on textual intermediates or raw visual annotations. Experimental results show that ConceptFormer achieves significant improvements in retrieval performance, with 16.7% and 22.1% relative gains in NDCG@10 over leading visual and OCR-based text retrieval baselines, respectively.
ConceptFormer reveals that adaptive latent concepts can dramatically enhance visual document retrieval by linking complex visual structures to semantic relevance without traditional textual dependencies.
Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7\% and 22.1\% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.
