Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper audits the integrity of whole-slide image (WSI) visual question answering (VQA) benchmarks, revealing significant data leakage at both patient and institutional levels that undermines the reported performance of vision-language models (VLMs). The authors document a staggering 92.3-100% overlap of training and test cases in TCGA-derived benchmarks, along with near-total overlap in staining-batch and scanner signatures, indicating that current evaluations fail to differentiate genuine multimodal reasoning from mere retrieval of memorized artifacts. The findings highlight the urgent need for improved benchmark construction and auditing practices to ensure the validity of performance claims in computational pathology.
Current WSI VQA benchmarks are so contaminated by data leakage that they misrepresent the true capabilities of vision-language models, with peak accuracies aligning with the most compromised datasets.
Recent vision-language models (VLMs) for computational pathology report striking zero-shot performance on whole-slide image (WSI) visual question answering (VQA) benchmarks. We audit these claims and find them fundamentally compromised by data leakage at two hierarchical levels: patient-level leakage, where slides from the same case appear in both training and test folds, and institutional-level leakage, where different cases nonetheless share staining-batch and scanner signatures through a common Tissue Source Site (TSS). By tracing canonical slide, case, and TSS identifiers across major public resources, we document case level train test overlaps of 92.3~100% on TCGA-derived benchmarks, together with near-complete TSS overlap. We further demonstrate that both leakage levels are linearly decodable from foundation-model feature space, that they induce a measurable accuracy gap between leaked and audit-clean cases on a published checkpoint, and that across multiple published WSI VLMs, peak reported accuracies concentrate on the most heavily contaminated benchmarks. Therefore, the current WSI VQA evaluation cannot distinguish genuine multimodal reasoning from nearest-neighbor retrieval over memorized institutional and patient-specific artifacts. Finally, we outline concrete recommendations for contamination-free evaluation. By addressing benchmark construction, provenance disclosure, and automated overlap auditing, we aim to guide future research toward verifiable claims of progress.