Search papers, labs, and topics across Lattice.
This paper presents an AI-driven system that evaluates human tutors by linking their training performance to real-life tutoring effectiveness, utilizing Generative AI (Gemini-2.5-pro) to analyze authentic tutoring transcriptions. The study involved 86 remote math tutors who showed a significant average learning gain of 7.4% after completing scenario-based lessons, with mixed-effects models revealing that training performance significantly predicted real-life transcript scores (effect size of 0.25 SD). Notably, the analysis indicated that averaging open response and multiple-choice performance during training was the most effective predictor of real-life tutor performance, with open responses being particularly predictive.
Training performance can significantly forecast real-life tutoring effectiveness, with open responses proving to be a stronger predictor than traditional assessments.
There exist numerous tutor training platforms. However, few provide AI-driven training and evaluation for human tutors based on real-life performance. We present an AI-driven system that assesses both open responses during training and authentic real-life tutoring. Unlike platforms that only assess learning through online training or simulations, our system utilizes Generative AI (Gemini-2.5-pro) to analyze transcriptions of authentic tutoring, measuring the transfer of tutor skills to real-life application. Human tutors instructing students remotely in math (N=86) completed six scenario-based lessons, averaging a significant 7.4% learning gain. Using mixed-effects models across 405 session-to-lesson pairs, we found that training performance significantly predicted real-life transcript scores with an effect size of 0.25 SD. Model comparison (AIC/BIC) indicated averaging open response and multiple choice performance during training predicted real-life tutor performance best, although open responses were comparatively more predictive. Exploratory analysis showed that after training, tutors were significantly more likely to encounter pedagogical opportunities to apply their skills (61.1% to 68.9%) and demonstrated higher execution quality within those opportunities (65.5% to 68.1%). Interrupted time series analysis suggested that these tutor improvements were part of a gradual trend over time rather than an immediate intervention effect of training. We illustrate an AI-driven method to link tutor training with real-life assessment. In doing so, we contribute open datasets, AI prompts, and scoring rubrics to support transparency and reproducibility.
CMU ML