Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the reliability of medical vision-language models (VLMs) in radiology, highlighting their failure to generalize across different acquisition domains and evaluation protocols. By isolating visual transfer and evaluating multimodal alignment using datasets like NIH ChestXray14 and CheXpert, the authors reveal that self-supervised visual initialization outperforms traditional supervised methods in cross-dataset transfer, while adversarial adaptation proves unstable. The findings underscore that models may appear competent in controlled settings but can exhibit significant transfer and alignment failures when subjected to distribution shifts, necessitating rigorous evaluation protocols.
Medical VLMs may seem reliable, but they can hide critical transfer and alignment failures that emerge under distribution shifts.
Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.