Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors conduct a controlled empirical analysis on the Kermany pediatric pneumonia benchmark across nine image classifiers to decouple real architectural progress from evaluation and pipeline artifacts. They find that pipeline choices—such as fine-tuning versus freezing backbones (0.044 AUROC shift) and threshold selection (0.090 balanced accuracy shift)—outweigh the performance spread across backbones (0.026 AUROC), while severe train-test split divergence allows a non-anatomical metadata classifier to reach 0.992 balanced accuracy on the training pool. These results expose how single-metric benchmark reporting masks pervasive shortcut learning and pipeline variance, motivating a grounded seven-item reporting framework for biomedical ML.
A model trained solely on file metadata—with zero medical imaging data—hits 0.992 balanced accuracy on a flagship pediatric x-ray benchmark, proving that evaluation artifacts and pipeline choices dominate architectural gains.
Biomedical machine learning papers often compress model performance into one headline number. That number can look like a property of the model even when it depends strongly on how the benchmark was evaluated. We study this problem on the widely used Kermany pediatric chest radiograph dataset using nine image classifiers and a controlled evaluation protocol. Under the same protocol, the eight pretrained backbones differ by only 0.026 AUROC. In contrast, changing whether the backbone is frozen or fine-tuned changes AUROC by 0.044 on average, and changing the decision threshold changes balanced accuracy by 0.090 on average. The official test split is also measurably different from the training pool: a partition classifier distinguishes them at AUC 0.697, rising to 0.898 for normal radiographs. Most strikingly, a classifier using only file properties, with no image anatomy, reaches 0.992 balanced accuracy within the training pool but falls to 0.496 on the official test split. Validation-fitted thresholds and calibration also transfer imperfectly. These results show that a high benchmark score can support different conclusions when the split, training policy, threshold, metric, calibration, and uncertainty are not communicated with it. We end with a seven-item reporting recommendation in which each item is tied to an effect measured in the study