Search papers, labs, and topics across Lattice.
This study investigates the internal representations of a frozen 3D chest vision-language model (Pillar-0) by probing its vision embeddings to identify how clinical findings are encoded. The authors reveal that each radiological finding is represented by a sparse set of approximately 10 channels, which not only matches the performance of full-feature classification but also significantly outperforms zero-shot text prompting. Additionally, their training-free concept channel probe (CCP) method demonstrates superior clinical efficacy and natural language generation metrics compared to existing models, indicating a generalizable property across different 3D medical encoders.
Radiological findings in frozen vision-language models are encoded by just 10 channels, revealing a surprising efficiency that challenges traditional assumptions about model complexity.
Large vision-language models are becoming increasingly dominant in 3D medical image interpretation, but we rarely knowwhichinternal units encode clinical findings orwherethat information lives in the representation. We first study this on a 3D chest vision-language model (Pillar-0) by probing its frozen vision embeddings. We show that (i) each radiological finding is encoded by asparseset of ~10 vision-encoder channels that match full-feature classification performance and far exceed a zero-shot text prompting; (ii) turning off the channels tied to one finding, that finding's score collapses while unrelated labels stay stable; and (iii) the same sparse probereplicateson an architecturally unrelated 3D abdominal VLM (Merlin) suggesting a general property of frozen medical encoders. Our training-free concept channel probe (CCP) method, paired with a corpus-derived report template, outperforms published CT-CHAT on clinical efficacy and NLG metrics (F1 0.549 vs. 0.184; BLEU 0.483 vs. 0.373) at 22x lower latency. Our results provide a clear, reproducible characterization of how frozen medical encoders represent findings, demonstrating direct applicability across models.
Stanford HAI