Search papers, labs, and topics across Lattice.
The authors introduce MedVL-SAM2, a unified 3D medical vision-language model designed for report generation, visual question answering (VQA), and multi-paradigm segmentation (semantic, referring, interactive). MedVL-SAM2 integrates image-level reasoning with pixel-level perception using a SAM2-based volumetric segmentation module for precise spatial reasoning in 3D medical imaging. The model is pre-trained on 3D CT image-text pairs and then jointly optimized with language-understanding and segmentation objectives, achieving state-of-the-art performance across the target tasks.
Finally, a single 3D medical vision-language model that nails both high-level reasoning (report generation, VQA) and fine-grained segmentation from language, point, or box prompts.
Recent progress in medical vision-language models (VLMs) has achieved strong performance on image-level text-centric tasks such as report generation and visual question answering (VQA). However, achieving fine-grained visual grounding and volumetric spatial reasoning in 3D medical VLMs remains challenging, particularly when aiming to unify these capabilities within a single, generalizable framework. To address this challenge, we proposed MedVL-SAM2, a unified 3D medical multimodal model that concurrently supports report generation, VQA, and multi-paradigm segmentation, including semantic, referring, and interactive segmentation. MedVL-SAM2 integrates image-level reasoning and pixel-level perception through a cohesive architecture tailored for 3D medical imaging, and incorporates a SAM2-based volumetric segmentation module to enable precise multi-granular spatial reasoning. The model is trained in a multi-stage pipeline: it is first pre-trained on a large-scale corpus of 3D CT image-text pairs to align volumetric visual features with radiology-language embeddings. It is then jointly optimized with both language-understanding and segmentation objectives using a comprehensive 3D CT segmentation dataset. This joint training enables flexible interaction via language, point, or box prompts, thereby unifying high-level visual reasoning with spatially precise localization. Our unified architecture delivers state-of-the-art performance across report generation, VQA, and multiple 3D segmentation tasks. Extensive analyses further show that the model provides reliable 3D visual grounding, controllable interactive segmentation, and robust cross-modal reasoning, demonstrating that high-level semantic reasoning and precise 3D localization can be jointly achieved within a unified 3D medical VLM.
