Search papers, labs, and topics across Lattice.
This paper addresses the challenge of standardizing raw, heterogeneous medical data for vision-language models (VLMs), which is crucial for their effective application in clinical settings. By constructing the Medical Data Standardization Benchmark (MDS-Bench) with 1,939 annotated tasks, the authors evaluate VLMs on their ability to process and convert fragmented medical data into structured image-text pairs. The findings reveal that even top-performing models like Gemini 3 Flash achieve only a 48.6% success rate, underscoring the need for advancements in data standardization for medical AI applications.
VLMs struggle with raw medical data, achieving only a 48.6% success rate in standardization, revealing a critical gap in their clinical applicability.
As vision-language models (VLMs) are increasingly applied to medical AI, existing benchmarks mainly focus on evaluating their diagnosis ability over given medical images and texts, implicitly assuming that standardized medical images, texts or question-answer pairs are already prepared. However, this assumption does not hold when we apply VLMs in real clinical practice, where medical data is often raw, heterogeneous, and fragmented across different sources. In this paper, we study this missing step, i.e., raw medical data standardization. Specifically, models are given raw dataset folders and evaluated on their ability to identify source formats, convert raw medical images into VLM-compatible visual inputs, extract relevant textual information, and organize the results into structured image-text pairs. To construct this Medical Data Standardization Benchmark (MDS-Bench), we manually annotate 1,939 raw medical data standardization tasks covering diverse clinical practice, radiology modalities, annotation formats, and directory layouts. Extensive experiments show that even the best performing VLMs, i.e., Gemini 3 Flash, achieve only 48.6% end-to-end success rate. Our research highlights raw medical data standardization as a critical bottleneck for medical AI diagnosis in real practice.
Stanford HAI