Search papers, labs, and topics across Lattice.
This paper introduces a clinician-centered pipeline designed for remote annotation and evaluation in ultrasound AI studies, addressing the limitations of existing platforms that focus primarily on dataset labeling. By facilitating blinded model comparisons and reproducible workflows, the pipeline enhances the clinical usability of AI systems in medical imaging. Validation through a fetal ultrasound segmentation study demonstrated moderate to strong agreement among raters of varying expertise, highlighting the pipeline's effectiveness in supporting clinician-centered evaluations.
Clinician-centered evaluations reveal that later active learning models are preferred, underscoring the importance of usability in ultrasound AI systems.
Clinician-centered evaluation is critical for validating medical AI systems, especially in ultrasound imaging where quantitative metrics do not always capture clinical usability. Existing medical image platforms primarily focus on dataset labeling. They lack integrated support for blinded model comparison and reproducible evaluation workflows. We present a clinician-centered pipeline for remote annotation and evaluation in ultrasound AI studies. The proposed pipeline uses a centralized server and lightweight browser interfaces to enable clinicians to perform annotation, blinded ranking, and review without local dataset downloads. The pipeline also supports multi-rater participation, centralized result aggregation, and automated statistical analysis. We validate the pipeline in a fetal ultrasound segmentation study with six raters spanning expert, generalist, and non-expert experience levels. The system automatically generated Spearman correlation, Kendall's $蟿$, and top-1 selection statistics. Results indicated moderate to strong agreement across experts and other groups. The blinded evaluation results showed a tendency for later active learning models to be preferred. These outcomes suggest that the pipeline can support clinician-centered annotation and reproducible human-\ac{AI} evaluation studies in ultrasound imaging. The proposed pipeline is available on \href{https://github.com/13204942/SonoRate}{GitHub}.
