Search papers, labs, and topics across Lattice.
This paper investigates the challenges of comparing similarity scores across different embedding models in Retrieval-Augmented Generation systems, which complicates model migration and threshold reuse. The authors introduce Synthetic Query Probing, a method that generates controlled query-chunk pairs from documents to analyze cross-model similarity behavior without requiring reference data. Their findings reveal that while models generally agree on rankings, their absolute scores are distorted, and the proposed score conversion functions, particularly isotonic regression, enhance threshold portability and calibration across models.
Systematic distortions in similarity scores across embedding models can be mitigated through a novel framework that improves threshold portability and model calibration.
Retrieval-Augmented Generation systems rely on similarity scores to retrieve relevant content, yet scores are not directly comparable across embedding models due to differing geometric properties, complicating model migration and limiting threshold reuse. We study how similarity scores can be related by learning mappings between score distributions rather than embeddings. We introduce Synthetic Query Probing, generating queries from documents to create controlled query-chunk pairs, enabling large-scale, reference-free analysis of cross-model similarity behavior. We evaluate the approach on multiple embedding configurations and learn score conversion functions using linear, isotonic, and quantile mappings. Experiments on SciFact and a proprietary corpus show that while models largely agree on rankings, their absolute scores exhibit systematic distortions. Learned mappings partially align these spaces and improve threshold portability, with isotonic regression performing best. Our results highlight the need for cross-model calibration and position Synthetic Query Probing as a scalable framework for analyzing embedding comparability.
