Search papers, labs, and topics across Lattice.
This study critically evaluates the effectiveness of pathology vision-language models (VLMs) by identifying significant shortcomings in their evaluation methods, particularly highlighting that visual evidence is not always necessary for accurate answers. The authors introduce PathBind, a comprehensive benchmark designed to address these evaluation gaps, consisting of 2,600 samples across various dimensions, which undergoes rigorous filtering and expert review. Results reveal that existing pathology VLMs demonstrate a notable disparity between their answer accuracy and their ability to effectively bind visual and semantic information, underscoring the need for improved evaluation metrics in this domain.
Current pathology VLMs can achieve high accuracy without visual input, revealing critical flaws in how we assess their multimodal capabilities.
Pathology vision-language models (VLMs) have recently progressed rapidly and are commonly evaluated by answer accuracy on pathology VQA benchmarks. However, we dig into current evaluations and identify three overlooked issues: 1) Visual evidence is not always necessary. For instance, Gemini-3-Pro achieves 53.5% average accuracy across 5 VQA benchmarks without any visual input. 2) Domain training can improve accuracy without proportional gains in visual binding. Compared with Qwen2.5-VL-7B, Patho-R1-7B exhibits a 5.8-point lower multimodal gain and a 3.7-point lower attention IoU. 3) Entity-level attention is diffuse and weakly query-specific. On PathVG, attention maps remain highly correlated across different entity queries. These issues can lead to substantial misjudgments of pathology VLMs'actual multimodal capabilities. To this end, we present PathBind, a benchmark comprising 2,600 samples: PathBind-VQA with 1,500 questions across six dimensions, PathBind-PTA with 600 questions from a private pathology teaching atlas, and PathBind-Grounding with 500 expert-curated region-level samples. Each component undergoes task-specific automated filtering and expert review to reduce textual shortcuts and improve entity-region correspondence. We evaluate 18 representative VLMs on VQA samples of PathBind and five existing pathology VQA benchmarks, and further evaluate 10 VLMs on PathBind-Grounding and PathVG. Results show that current pathology VLMs still exhibit a substantial gap between answer-side performance and visual-semantic binding.
