Search papers, labs, and topics across Lattice.
MedPixel is a novel unified pixel-language model designed to enhance medical image understanding by bridging the gap between clinical language and visual reasoning with pixel-level grounding. It leverages a new dataset, MedPLG-440K, which consists of approximately 440,000 pixel-language task samples synthesized without external LLM annotation, addressing the supervision mismatch in existing models. The model demonstrates strong performance across a variety of tasks, including explicit grounding and medical visual question answering, while also exhibiting robustness to imperfect spatial prompts and effective zero-shot transfer capabilities.
MedPixel achieves state-of-the-art performance in medical image reasoning by seamlessly integrating pixel-level grounding with clinical language understanding.
Reliable medical image understanding requires models to connect clinical language and visual reasoning with pixel-level grounding. Yet medical vision-language models often lack precise localization, whereas medical segmenters typically rely on explicit target categories or precise spatial prompts. This divide is reinforced by a supervision mismatch: segmentation datasets provide precise masks but little language supervision, whereas medical vision-language data rarely pair language with dense spatial annotations. To address this gap, we present MedPixel, a unified medical pixel-language model built around a shared language--mask interface. To provide scalable supervision, we introduce MedPLG-440K, comprising approximately 440K pixel-language task samples constructed through a clinically motivated synthesis process without external LLM annotation. MedPixel is trained with joint multi-task supervised fine-tuning followed by Pixel-Level Preference Optimization, which uses ground-truth masks as offline verifiers to derive response preferences from mask quality. MedPixel supports a broad spectrum of tasks spanning explicit grounding, implicit reasoning, spatial interaction, grounded explanation, and medical VQA. Across this task spectrum, MedPixel achieves strong performance in both pixel-level prediction and response generation, together with effective zero-shot transfer to external grounding benchmarks and robustness to imperfect spatial prompts. Code and model checkpoints will be released at https://github.com/yhy-whu/Medpixel.
