Search papers, labs, and topics across Lattice.
This paper investigates the relationship between embedding magnitudes and semantic properties in contrastive embedding models, revealing that discarded norms correlate with concept specificity and human uncertainty. By analyzing optimization dynamics, the authors derive a formula that shows how embedding length encodes semantic information as a byproduct of training. The findings suggest that these norms can serve as effective calibration tools in retrieval tasks, providing a theoretical basis for previously observed heuristics in the field.
Embedding magnitudes, often ignored in contrastive models, actually encode critical semantic information that can enhance retrieval tasks.
Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. However, surprisingly, empirical studies reveal that despite this, these "discarded" norms seem to correlate with semantic properties such as concept specificity, token frequency, and human uncertainty. In this work, we provide a formal theoretical framework explaining this phenomenon. By analyzing the optimization dynamics, we derive an analytic formula demonstrating that embedding length naturally encodes this information as a byproduct of the training process. We also show how this gives rise to signals that can serve as "free" calibration tools in specific models and retrieval tasks, providing a grounded explanation for a previously heuristic observation.
