Search papers, labs, and topics across Lattice.
This study investigates the influence of evaluation annotations on pulmonary embolism (PE) segmentation performance, comparing it to the effects of model training changes using two pretrained nnU-Net models. By analyzing 149 voxel-annotated CT cases, the researchers found that refined annotations significantly improved segmentation accuracy, with mean Dice Similarity Coefficients (DSC) increasing by 0.143 for nnU-Net-A and 0.188 for nnU-Net-B, while changes in training dataset composition yielded only a minor improvement of 0.028. The findings highlight that the quality of annotations can have a greater impact on segmentation performance than the choice of model architecture or training data, establishing a human-referenced benchmark for future research.
Refined annotations can boost PE segmentation performance more than changes in model training, challenging assumptions about model superiority.
Purpose: To quantify how evaluation annotations influence measured pulmonary embolism (PE) segmentation performance relative to model training changes, and to establish a human-referenced framework. Materials and Methods: This retrospective study screened 166 voxel-annotated CT pulmonary angiography cases from CADPE (n=91), FUMPE (n=35), and READ (n=40); 149 were included. A primary rater annotated PE by protocol, and a senior thoracic radiologist reviewed and revised all segmentations. Three additional raters at three centers annotated a 15-case subset. The label effect was measured by evaluating two pretrained nnU-Net models (nnU-Net-A, nnU-Net-B) against original and refined annotations. The model effect was measured by comparing the same architecture trained on different dataset combinations with annotations fixed. The benchmark model (nnPE) was trained with leave-one-dataset-out and pooled five-fold cross-validation. Four metric categories were analyzed with case-paired Wilcoxon signed-rank tests, Benjamini-Hochberg correction, and bootstrap 95% CIs. Results: Changing only the annotation increased mean DSC by 0.143 (0.122-0.166) for nnU-Net-A and 0.188 (0.163-0.213) for nnU-Net-B (both P < .001), whereas changing training-dataset composition changed DSC by 0.028. The label effect exceeded the model effect on CADPE and FUMPE and was 0.045 on READ. Within-mask attenuation SD fell in all three datasets after re-annotation (all P < .001). nnPE reached DSC 0.72 +/- 0.22 on pooled cross-validation but scored below all four annotators across 52 paired comparisons (all corrected P < .05). Conclusion: Evaluation annotations affected measured PE segmentation performance at least as much as model training choices. A human-referenced evaluation framework is publicly available for future study.
