Search papers, labs, and topics across Lattice.
This paper introduces lightweight fine-tuning as a method to optimize late interaction models for multi-vector compression, addressing the challenge of high storage and memory costs associated with token vectors. By employing minimal pooling-aware training with k-means, the authors demonstrate significant improvements in retrieval accuracy across various compression levels, outperforming traditional inference-only pooling methods. The proposed approach achieves an impressive 83% vector compression rate while maintaining accuracy, showcasing its potential for practical deployment in resource-constrained environments.
Achieving an 83% reduction in vector count without sacrificing retrieval accuracy could revolutionize the deployment of late interaction models in real-world applications.
Late interaction models have shown strong generalization capabilities, often outperforming much larger dense embedding models. One challenge to their widespread deployment is the large number of token vectors they produce per document and the associated storage and memory costs. Pooling tokens at inference time has shown great promise to reduce the vector count with limited effects on retrieval accuracy. Large-scale pooling-aware training has demonstrated even more impressive results at high compression rates. We propose lightweight fine-tuning as a practical alternative and find that even minimal pooling-aware training with k-means yields broad gains over inference-only pooling, shows evidence of transfer across pooling methods and datasets, and - with multi-factor training - produces a single model effective across different compression levels. Our strongest model outperforms the unpooled baseline on BEIR SciFact across pool factors 1-6, implying a vector compression rate of 83% at no cost to retrieval accuracy.
