Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
FGR-ColBERT distills fine-grained relevance signals from a 27B LLM into a ColBERT retrieval model by training the ColBERT embeddings to predict token-level relevance scores. This allows the smaller FGR-ColBERT (110M parameters) to achieve a token-level F1 score of 64.5 on MS MARCO, surpassing the 62.8 achieved by the larger Gemma 2 (27B) model. FGR-ColBERT maintains retrieval effectiveness and introduces only a marginal increase in latency compared to the original ColBERT.
A 110M parameter ColBERT model can now outperform a 27B parameter LLM in identifying fine-grained relevant tokens within retrieved documents, while remaining efficient for retrieval.
Document retrieval identifies relevant documents but does not provide fine-grained evidence cues, such as specific relevant spans. A possible solution is to apply an LLM after retrieval; however, this introduces significant computational overhead and limits practical deployment. We propose FGR-ColBERT, a modification of ColBERT retrieval model that integrates fine-grained relevance signals distilled from an LLM directly into the retrieval function. Experiments on MS MARCO show that FGR-ColBERT (110M) achieves a token-level F1 of 64.5, exceeding the 62.8 of Gemma 2 (27B), despite being approximately 245 times smaller. At the same time, it preserves retrieval effectiveness (99% relative Recall@50) and remains efficient, incurring only a ~1.12x latency overhead compared to the original ColBERT.