Search papers, labs, and topics across Lattice.
This paper introduces SCALPEL, a Semantic Cross-modal Alignment framework that enhances medical vision-language representation by addressing key limitations of existing models. By converting a generative LLM into an isotropic encoder through Clinical Report Contrastive fine-tuning and employing an Anatomy-Negation Aware Objective, SCALPEL effectively mitigates issues such as representational collapse and medical hallucinations. Experimental results on MIMIC-CXR, CheXpert, and IU X-Ray datasets show that SCALPEL achieves state-of-the-art performance in cross-modal retrieval, zero-shot disease classification, and medical visual question answering.
SCALPEL redefines medical vision-language representation by eliminating hallucinations and enhancing context understanding, achieving state-of-the-art results across multiple benchmarks.
Vision-language pre-training (VLP) serves as a cornerstone for medical multimodal representation learning. However, existing medical VLP frameworks are often constrained by the limited context windows and shallow representational capacities of lightweight text encoders when processing lengthy, terminology-dense clinical reports. While integrating medical large language models (LLMs) offers unprecedented clinical reasoning capabilities, it introduces three major bottlenecks: (i) the anisotropic representational collapse of generative LLMs under standard contrastive objectives, (ii) the prohibitive memory overhead of joint end-to-end training with large batch sizes, and (iii) the medical hallucinations induced by vanilla contrastive losses that ignore fine-grained anatomical laterality and negation modifiers. To address these challenges, we propose \textbf{SCALPEL}, a \textbf{S}emantic \textbf{C}ross-modal \textbf{A}lignment framework via \textbf{L}LM-\textbf{P}owered \textbf{E}ncoder \textbf{L}earning. First, Clinical Report Contrastive fine-tuning converts a generative LLM into an isotropic encoder via domain-specific clinical text adaptation. Second, an asymmetric alignment strategy leverages offline feature caching to enable efficient training. Critically, we formulate an Anatomy-Negation Aware Objective that explicitly penalizes mismatched image-text pairs involving laterality confusion or false negations. Extensive experiments across MIMIC-CXR, CheXpert, and IU X-Ray benchmarks demonstrate that SCALPEL achieves state-of-the-art performance in cross-modal retrieval, zero-shot disease classification and medical visual question answering.
