Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper conducts a forensic reproducibility audit of a chest-radiograph vision-language model (VLM) benchmark, revealing significant discrepancies between the intended protocol and the released artifacts. The audit uncovered issues such as improper image rendering, loss of dataset split membership, and incomplete report generation, which ultimately led to a substantial change in statistical outcomes. As a result, the authors retract previous performance claims and propose machine-verifiable controls to ensure future reproducibility in medical AI benchmarks.
The audit reveals critical flaws in a widely used radiology VLM benchmark, prompting a retraction of its performance claims and highlighting the need for stringent reproducibility standards.
Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases. Agreement across these artifacts is usually assumed rather than tested. We performed a retrospective forensic reproducibility audit of a preserved chest-radiograph vision-language model (VLM) pilot; no model was called again and no image or report was newly annotated. We traced prompt bindings, DICOM metadata, output completeness, label extraction, matched analyses, and release propagation. Of 300 planned model-prompt calls, 297 yielded nonempty reports. Sixty Claude calls labeled A/B were executed with the same C prompt. The 30 studies represented 28 patients. Four MONOCHROME1 images were rendered without required polarity inversion, dataset split membership was not retained, and the unvalidated extractor truncated five reports to 4000 characters. Reconstructing one common cohort of 369 complete case-finding blocks changed Cochran's Q from 154.73 to 182.29. Of 45 McNemar comparisons, 27 had unadjusted p<0.05 and 20 remained below 0.05 after Holm adjustment. These values describe only the archived automated-label matrix; they do not recover the intended prompt comparison or establish clinical performance. We withdraw the original performance, ranking, prompt-effect, and clinical claims and specify machine-verifiable controls for cohort, DICOM rendering, prompt and model identity, call status, annotation provenance, keyed analysis, and derived artifacts.