Search papers, labs, and topics across Lattice.
This study investigates the representational convergence of medical image encoders, revealing that self-supervised learning significantly drives this convergence more than clinical supervision. By analyzing 18 image and 7 text encoders across diverse datasets, the authors found that self-supervised encoders achieved a 40.4% alignment on chest radiography, while label-supervised and image-text encoders showed much lower alignment. The findings suggest that interoperability in medical imaging should be designed around self-supervised objectives rather than relying on scale or clinical supervision.
Self-supervised learning outperforms clinical supervision in aligning medical image representations, challenging assumptions about model interchangeability.
Medical image encoders from different groups are increasingly treated as interchangeable, on the assumption that scale and clinical supervision concentrate their representations onto a shared structure. Whether this convergence is real, what produces it, and whether it is clinically usable are untested, and the similarity measures behind such claims are fragile. We present a controlled dissection across 18 image and 7 text encoders, all open-weight and run locally, spanning 7M to 27B parameters and five imaging modalities, including 650,982 chest radiographs from six datasets. To isolate cause, we train encoders that vary only the objective under fixed data, architecture, and scale, and reproduce the effect in a synthetic model. Convergence is modest but above a random floor, driven by the self-supervised objective, not clinical supervision: matched self-supervised encoders aligned most (40.4% on chest radiography), with label-supervised (21.1%) and image-text (3.3%) far lower, and did not grow with size (Spearman 0.302, p=0.223) or capability. It is within-modality, does not reach clinical language, and does not reproduce how radiologists judge case similarity. Yet a linear classifier transfers across encoders and to five held-out hospitals, retaining about 85% of within-encoder performance. Convergence in medical imaging is therefore set by the pretraining objective, not inherited from scale or clinical supervision. Interoperability is accordingly something to design for through that objective, and to validate where the shared geometry is weakest, across patient subgroups and against clinical judgment.
