Search papers, labs, and topics across Lattice.
This paper introduces a novel vision-language pre-training framework specifically designed for 3D computed tomography (CT) imaging, addressing limitations in existing methods related to visual encoding and semantic alignment. By employing a CNN-ViT hybrid encoder, a disease-level contrastive learning mechanism, and a diagnosis-aware prompt strategy, the model effectively captures local anatomical details and enhances the alignment of disease-specific semantics with visual features. The proposed approach achieves state-of-the-art performance on multiple benchmarks, significantly improving diagnostic accuracy and demonstrating strong applicability in clinical settings.
A tailored vision-language framework boosts diagnostic accuracy on 3D CT imaging by over 5% while enhancing zero-shot reliability through innovative semantic alignment techniques.
Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN-ViT hybrid encoder that replaces ViT's patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4% AUC, +5.1%) and Rad-ChestCT (75.4% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach.
DAMO