Search papers, labs, and topics across Lattice.
This study evaluated the inter-rater reliability of a three-stage retrospective medical record review process for identifying adverse events in 360 patients across three medical institutions in South Korea. The study found moderate inter-rater reliability in both the initial screening stage (kappa 0.558) and the subsequent adverse event review stage (kappa 0.421), suggesting that involving diverse healthcare professionals in the review process is feasible. The authors conclude that while diversification of reviewers is supported, rigorous procedures to enhance interrater reliability are essential.
Diversifying healthcare professionals involved in retrospective medical record reviews for adverse events is feasible, but standardized training and interim assessments are needed to improve inter-rater reliability.
BACKGROUND Training and engaging various healthcare professionals in the review process is important for improving the feasibility and utility of retrospective medical record reviews in identifying adverse events. This study aimed to evaluate the inter-rater reliability of retrospective medical record review for identifying adverse events and to examine the feasibility of involving diverse types of healthcare professionals as reviewers. METHODS A three-stage retrospective medical record review was conducted at three medical institutions following the methodology of the Korean Patient Safety Incident Inquiry conducted in South Korea in 2019. In the first stage, the presence or absence of 41 predefined screening criteria-nationally used in South Korea to screen for potential adverse events-was assessed by two nurses or by one nurse and one health information manager in 120 randomly selected patients at each institution. In some cases, external reviewers participated alongside the internal reviewers. In the second stage, if any screening criteria were identified, two physicians or one nurse and physician evaluated the occurrence of adverse events. In the third stage, the occurrence and characteristics of adverse events were finalised. Inter-rater reliability was assessed for the first and second stage using percentage agreement and kappa statistics. RESULTS Of 360 reviewed patients across three institutions, 139 (38.6%) were identified by screening criteria, 39 underwent detailed adverse event review, and 47 adverse events were ultimately confirmed by the final committee. The overall agreement rate in the first-stage review was 81.4% (including all negative cases), with a kappa value of 0.558 (95% confidence interval [CI]: 0.467-0.649). In the second-stage review, the agreement rate was 82.0%, with a kappa value of 0.421 (95% CI: 0.223-0.588). The agreement between nurses and health information managers in the first stage was 82.9% (kappa = 0.600, 95% CI: 0.494-0.705). For internal and external reviewers, it was 75.0% (kappa = 0.446, 95% CI: 0.303-0.589). In the second stage, the agreement between physicians and nurses was 82.9% (kappa = 0.600, 95% CI: 0.295-0.848). CONCLUSIONS This study provides evidence supporting the diversification of reviewer professions and types at each stage of retrospective medical record reviews to identify adverse events. However, establishing rigorous procedures to enhance interrater reliability, such as training in medical record review methods and conducting interim assessments during the review process, is essential.
