Search papers, labs, and topics across Lattice.
This paper introduces MarginMerge, a novel compression method for multi-vector visual document retrievers that prioritizes query-relevant coverage over independent patch salience. By selecting coverage-aware anchors and clustering document patches, MarginMerge significantly reduces the size of stored document vectors while maintaining high retrieval performance, achieving 97-99% preservation of nDCG@5 with a 90-95% reduction in vector storage. Remarkably, this method not only retains effectiveness across multiple datasets but also transfers seamlessly to unseen datasets and varying retention ratios without requiring retraining.
Achieving up to 99% retrieval effectiveness with a 95% reduction in storage, MarginMerge redefines how we compress visual document retrievers.
Multi-vector visual document retrievers such as ColPali and ColQwen achieve strong retrieval by storing fine-grained patch embeddings, but this produces large indexes and costly late-interaction scoring. We argue that effective compression should preserve query-relevant coverage, meaning the diverse document regions that may become the strongest MaxSim match across queries, rather than selecting patches independently by salience. This view also explains why dense rendered pages are easier to compress than natural images. We introduce MarginMerge, a compression method for frozen multi-vector retrievers. It selects coverage-aware anchors, clusters document patches, and uses a lightweight shared network to synthesize one representative per cluster. Compression is performed once during indexing, while retrieval keeps the standard MaxSim interface. Across six datasets on both ColQwen2.5 and ColPali, MarginMerge achieves the highest matched query-agnostic average at 5% and 10% vector retention. Compared with the uncompressed index using the same backbone, it preserves between 97% and 99% of average nDCG@5 while reducing stored document vectors by between 90% and 95%. At 5% retention, it also reduces ranking flips relative to geometric merging on all six ColQwen2.5 datasets by approximately 41% on average. The same model transfers to unseen datasets and retention ratios without retraining.
