Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study conducts a thorough comparative analysis of label-free metrics for evaluating representation quality in deep neural networks, categorizing existing metrics into three families and establishing their interconnections. Through controlled synthetic experiments, the research assesses the sensitivity of spectral metrics and evaluates their effectiveness across 260 vision models on six diverse datasets, including various classification and recognition tasks. The findings reveal that intrinsic dimensionality (ID) serves as the most consistent predictor of representation quality, though its reliability varies significantly based on the architecture class and training objective employed.
Intrinsic dimensionality emerges as the most dependable metric for representation quality, but its effectiveness is heavily influenced by model architecture and training objectives.
We present a comparative study of label-free metrics for assessing the quality of representations in deep neural networks to understand their reliability under a wide variety of configurations. We group existing label-free metrics into three families based on their construction and analytically establish connections between metrics within the same family. We then characterise the sensitivity of spectral metrics through controlled synthetic experiments. Finally, all label-free metrics are evaluated against downstream task accuracy across a diverse set of 260 vision models on six datasets spanning generic object classification, fine-grained object classification, scene recognition and geospatial task, stratifying results by architecture class and training objective. We find that intrinsic dimensionality (ID) is the most reliable predictor among the metrics considered. However, the reliability of all metrics, including ID, is moderated by architecture class and training objective. Our results provide a clearer understanding of what label-free representation quality metrics measure, when they are reliable, and how to interpret them in practice.