Search papers, labs, and topics across Lattice.
This study investigates the vulnerability of vision-language models (VLMs) in detecting synthetic medical images when contextualized with structured records and metadata. By reformulating the detection task as an audit of multimodal robustness, the authors reveal that VLMs can significantly alter authenticity judgments based solely on changes in accompanying text, highlighting a critical flaw in current evaluation methods. The introduction of a paired benchmark allows for a systematic assessment of this issue across various imaging modalities, underscoring the need for improved robustness in real-world applications.
VLMs may misjudge the authenticity of medical images based on metadata alone, leading to potentially dangerous diagnostic errors.
With the rapid adoption of generative AI, synthetic medical images pose growing risks, including diagnostic deception and insurance fraud. Although prior work has explored vision-language model (VLM)-based synthetic image detection, these evaluations typically consider images in isolation. In clinical practice, however, images are interpreted alongside structured records and metadata, and VLMs are increasingly deployed under joint image-record inputs. We uncover a previously underexamined multimodal vulnerability: when given both modalities, VLMs may overweight record context in authenticity judgments, such that the same image receives different predictions solely due to changes in its accompanying text. This raises concerns about robustness in real-world deployment. To systematically characterize this effect, we reformulate synthetic medical image detection as an audit of multimodal robustness at the image-record interface and introduce a paired benchmark that holds the image fixed while swapping controlled metadata variants. Across multiple imaging modalities, we evaluate diverse open-weight and frontier API VLMs and quantify how metadata alone shifts authenticity predictions. Our benchmark provides a standardized tool for assessing and improving multimodal robustness beyond image-only settings. The code is available at https://github.com/chiuhaohao/Beyond-Visual-Forensics.
