Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces Quality Action Assurance (QAA), a multimodal framework designed to enhance the reliability of scoring in Virtual Reality (VR) pediatric Objective Structured Clinical Examinations (OSCEs) by verifying examiner claims against actual events. By integrating a constrained temporal action alignment model with a large language model, QAA effectively localizes actions and attributes them to the correct sources, achieving high precision and recall in error detection. The results demonstrate a significant improvement in factual correctness, raising it from 39.2% to 79.2%, thereby addressing biases in examiner assessments.
QAA not only boosts factual correctness in OSCE evaluations but also reveals the hidden biases in examiner scoring, transforming how clinical competence is assessed.
Objective Structured Clinical Examinations (OSCEs) are the gold standard for assessing clinical competence, yet scoring remains vulnerable to examiner subjectivity, fatigue, and cognitive bias. Standard examiner validation via inter-rater statistics lacks explanatory power regarding the source of errors, as it neither analyzes examiner reasoning nor verifies examiner claims against actual events. Thus, we introduce Quality Action Assurance (QAA), a multimodal framework that verifies examiner claims in Virtual Reality (VR) pediatric OSCEs by comparing actions claimed by examiners against the true sequence of events, constructed from video, VR logs, and actor data. QAA combines a constrained temporal action alignment model, which performs action localization and actor source attribution, with a large language model that extracts examiner claims and checks them against the record. Across a 5-fold cross-validation, QAA achieves 99.2% $\pm$ 0.7% Actor F1 and 93.4% $\pm$ 1.9% W@16 for temporal alignment. Overall, QAA detects examiner errors with 70.0% precision and 76.7% recall, improving factual correctness from 39.2% to 79.2%, enabling fairer OSCE assessment.