Search papers, labs, and topics across Lattice.
Stanford HAIColumbiaDepartments of 1 PsychologyElectrical EngineeringNeuroscienceJun 17, 2026arXiv:2606.19603This study extends the understanding of linear probes in interpretability research by comparing traditional cosine similarity with Mahalanobis cosine similarity (MCS), which accounts for test data covariance. The authors demonstrate that MCS can predict out-of-distribution (OOD) AUROC scores with high accuracy (R^2 = 0.98) across various models and domains, establishing a linear relationship under specific conditions. Additionally, they provide theoretical insights into when this linearity may break down, offering a robust framework for evaluating probe performance.
MCS can predict OOD performance with astonishing accuracy, revealing a hidden linearity that traditional methods miss.
Linear probes are widely used in interpretability research and often compared by cosine similarity. The Mahalanobis cosine similarity (MCS) between two directions, which reweights the inner product by test data covariance, is a natural task-aware refinement. Ying et al. (2026) report that a probe's MCS to a reference probe trained on the out-of-distribution (OOD) data near-perfectly linearly predicts the probe's OOD AUROC (R^2 = 0.98). Here, we extend this empirical finding across models, layers, and concept domains, and prove this general phenomenon in closed form: For balanced classes whose projections are Gaussian, OOD AUROC and MCS to the reference probe are linear because both are sigmoid-shaped functions of the probe's signal-to-noise ratio (SNR) on the test data. The theory also predicts when this linearity fails, which we verify empirically. MCS offers a theoretically grounded and empirically effective alternative to Euclidean cosine similarity for comparing linear probes.
