Search papers, labs, and topics across Lattice.
This paper introduces ColSNAP, a novel training method that enables multi-granularity visual document retrieval by generating a nested hierarchy of compression levels from a backbone's patch grid. By employing Spatial Nested Average Pooling, the method allows for simultaneous training of multiple tiers of compression, which can be flexibly configured at indexing time to optimize the accuracy-storage trade-off. The results show that models trained with ColSNAP maintain nearly full-resolution retrieval performance even under significant compression and can be effectively adapted across various late-interaction retrieval architectures.
ColSNAP allows for flexible, multi-level compression in visual document retrieval without sacrificing retrieval performance, adapting seamlessly across different architectures.
Multi-modal late-interaction retrievers achieve strong retrieval on visually rich documents by representing each page as per patch embeddings and matching at the token level. However, this approach incurs high storage costs. Existing compression methods typically fix a single compression level at indexing time, limiting flexibility. We present ColSNAP (Spatial Nested Average Pooling)1, a training method that generates a nested hierarchy of compression levels directly from a backbone's patch grid. By spatially pooling patch embeddings into pro- gressively coarser tiers and training all tiers simultaneously, a single model learns to support retrieval at multiple compression levels without architectural changes. Crucially, a single encoding pass yields every tier, enabling the accuracy-storage trade-off to be configured at indexing time to match avail- able storage budgets, rather than being fixed during training. We demonstrate that models trained using ColSNAP maintain near full-resolution retrieval performance under substantial compression and that ColSNAP transfers effectively across multiple late-interaction backbones, and achieves most of its improvements via a lightweight adaptation stage applied to a pre-trained retriever.
