Search papers, labs, and topics across Lattice.
This paper introduces ConQuer, a novel image-text pretraining method that enhances CLIP-style global alignment by incorporating localized alignments for concept-specific sections in radiological reports. By employing cross-attention queries that pool matching image features without requiring segmentation masks, ConQuer enables a more nuanced understanding of complex medical images. The resulting 3D CT foundation model, Jolia, significantly outperforms traditional CLIP baselines in tasks such as findings classification and report generation, establishing new benchmarks in the field.
Jolia not only sets a new state of the art in 3D medical imaging but also reveals that localized concept alignment can dramatically enhance model performance in complex tasks.
Vision-language contrastive pretraining has become the dominant recipe for 3D medical foundation models, leveraging the large volumes of paired scans and reports produced in clinical practice. However, medical images usually span dozens of organs, and radiological reports are much longer than typical natural image captions and are composed of multiple structured sections. CLIP-style pretraining compresses this structure by encoding each modality into a single global token, at the risk of losing important details. We introduce ConQuer (Concept Queries), an image-text pretraining method that augments CLIP's global alignment with a set of localized alignments, one per concept. ConQuer splits the report into concept-specific sections and learns cross-attention queries that pool the matching image features without using any segmentation mask or spatial supervision. Contrastive learning is then applied independently for each concept. Concepts can be any unit of semantic localization; here, they are anatomical regions, one query per organ or gross body region. As a byproduct, each query learns attention maps focused on its concept, providing built-in spatial interpretability. We use ConQuer to train Jolia, a 3D CT foundation model on chest and abdominal CT. Jolia consistently outperforms a CLIP baseline on findings classification, report generation, and cross-center transfer, and sets a new state of the art across multiple public benchmarks. Jolia's weights will be released upon acceptance.
