Search papers, labs, and topics across Lattice.
This paper introduces SaMer, an object-aware token merging framework that compresses image-side tokens into representative centroids while maintaining the integrity of late-interaction interfaces in vision-language retrieval. By leveraging object annotations during training, SaMer effectively reduces the number of tokens by over 93% and storage requirements by 16.09 times, all without needing ground-truth bounding boxes at inference. The method not only improves retrieval accuracy on datasets like Flickr30K and MSCOCO but also enhances phrase-level grounding by preserving critical object evidence necessary for future queries.
Merging image tokens intelligently can cut storage costs by over 16 times while boosting retrieval accuracy, challenging the notion that less data always means less context.
Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into $K$ representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With $K=64$, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by $16.09\times$, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.
