Search papers, labs, and topics across Lattice.
This paper introduces MedUP, a Medical Vision-Language Model (Med-VLM) that integrates visual perception and language understanding within a unified token space, addressing the limitations of existing models that separate these functions. By employing a novel region tokenizer, UniMedTok, which encodes visual masks as discrete tokens, MedUP achieves seamless interleaving of visual and textual information. Experimental results indicate that MedUP significantly outperforms existing Med-VLMs and remains competitive with specialized segmentation models across various tasks, highlighting the effectiveness of unified modeling in medical applications.
MedUP reveals that integrating visual perception and language understanding in a single token space can dramatically enhance performance in medical vision-language tasks.
Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling.
