Search papers, labs, and topics across Lattice.
This study systematically examines how the choice of evaluation references impacts the performance and ranking of machine learning models for chest X-ray (CXR) analysis, revealing that commonly used report-derived labels and image quality metrics often misalign with clinical judgment. By collecting paired expert labels and assessing various models, including ResNet and vision-language models like MedKLIP, the authors demonstrate that changing label sources can lead to significant variations in performance estimates and model rankings. The findings underscore the necessity of carefully selecting evaluation references to ensure clinical validity in CXR machine learning applications.
Evaluation reference choices can drastically alter model performance and rankings, challenging the reliability of current CXR machine learning assessments.
Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment. However, commonly used report-derived labels for pathology classification or generic image quality metrics for reconstruction may not reliably reflect clinical judgment. We systematically investigate how evaluation-reference choices affect model performance and ranking in both pathology classification and image quality assessment (IQA). To enable controlled comparison across evaluation references, we collected paired expert image- and report-derived labels for thoracic findings from a clinical cohort at Cambridge University Hospitals (CUH) and curated a subset of the public MIMIC-CXR dataset, along with expert ratings of diagnostic image quality. We show that for supervised image classifiers (ResNet, DenseNet), several zero-shot and fine-tuned vision-language models (e.g., MedKLIP, GLoRIA, and ConVIRT), changing the label source leads to substantial differences not only in performance estimates but also in model rankings. In parallel, alignment of IQA measures with expert judgment depends heavily on the choice of measure, and commonly used IQA metrics such as SSIM and PSNR often fail to align with expert assessments of diagnostic usability. Our results demonstrate that evaluation choices are crucial: they can determine which models and methods appear best and are therefore selected for further development or deployment. The selection of evaluation references should therefore be treated as a central component of clinical validity in CXR machine learning, and justified with respect to the pathology, imaging task, and intended downstream clinical use.
