Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of a compact retriever that learns ranking behavior from score vectors, bypassing the need for access to teacher hidden states. By employing a row-centered score-vector objective and a memory-efficient PairMSE loss, the authors demonstrate that their distillation protocol can recover up to 50% of the performance gap between a large teacher model and a smaller student model. The resulting 0.6B student model significantly outperforms the teacher in terms of speed, achieving 4.7脳 faster query encoding and 9.7脳 faster document encoding, although external-transfer performance remains inconsistent.
Compact retrievers can achieve nearly half the performance of larger models while being up to 9.7脳 faster, challenging the assumption that bigger is always better in retrieval tasks.
Large embedding models improve retrieval quality, but serving large encoders online is expensive. We study whether a compact retriever can learn teacher ranking behavior from score vectors without access to teacher hidden states. The student trains on rows built from ground-truth positives and negative candidates produced by our data generation pipeline; we evaluate student-teacher hard-negative mining separately as an extension. We use a row-centered score-vector objective, a memory-efficient implementation of uniform all-pairs PairMSE loss. On a fixed eight-task evaluation panel, our distillation protocol recovers up to 50\% of the base-to-teacher gap. The distilled 0.6B student is 4.7$\times$ faster for query encoding and 9.7$\times$ faster for document encoding than sequential online teacher fusion. External-transfer performance after distillation remains mixed, so our evidence supports compression of teacher rankings under matched retrieval protocols.
