Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
ColNanoVDR establishes a document-free query distillation framework for multi-vector visual document retrieval by aligning student and teacher query representations via entropic optimal transport with learned token weights. By theoretically bounding the MaxSim score difference on arbitrary pages solely through query token alignment, the approach eliminates the terabyte-scale page caching and document-encoding overhead historically required by score distillation. Evaluated across ViDoRe v1–v3 benchmarks, 149M text-only student encoders retain approximately 95% of their multi-billion-parameter VLM teachers' NDCG@5 while delivering up to 26x faster query encoding and consuming 12.6x less cached training data.
Distilling multi-billion-parameter multi-vector retrievers no longer requires caching terabytes of document tokens: entropic optimal transport allows a lightweight 149M text model to capture 95% of VLM teacher performance at 26x query speed using queries alone.
Multi-vector retrievers built on vision-language models lead visual document retrieval (VDR), but they run a multi-billion-parameter query encoder on every search. Distilling this encoder into a small student that queries the teacher's existing index would remove the bottleneck. The standard recipe, however, matches the teacher's MaxSim scores and so requires encoding and caching every training page, which can reach terabytes of page tokens. NanoVDR avoids pages entirely by training on the teacher's query embeddings alone, but only for single-vector retrievers. We present ColNanoVDR, to our knowledge the first framework to bring this document-free distillation to multi-vector VDR. Its objective, OTW (Optimal Transport with Learned Weights), aligns the student's query tokens with the teacher's by entropic optimal transport, with a learned weight for each student token, and needs no correspondence between the two tokenizations. We prove that the resulting alignment cost bounds the MaxSim score difference on every page. Distilled from five state-of-the-art teachers, the 149M text-only students retain about 95% of their teachers' NDCG@5 on ViDoRe v1-v3 while encoding queries up to 26x faster. Under identical training, OTW matches score distillation while encoding no page and reading 12.6x less cached teacher data.