Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study leverages a few-shot text classifier to map student questions directed at conversational AI teaching assistants to specific curriculum topics, utilizing a prerequisite knowledge graph derived from GPT-4. Evaluated on 1,340 interactions from graduate students, the classifier achieved an 80.0% accuracy across 43 labels, revealing a significant correlation between the volume of questions on specific topics and students' self-reported difficulties. The findings suggest that analyzing conversational AI logs can effectively identify knowledge gaps in students, offering instructors valuable insights into which topics require further focus.
Conversational AI logs can reveal critical knowledge gaps in students, correlating question patterns with actual learning difficulties.
Large online courses generate thousands of student questions directed at conversational AI teaching assistants, yet these interaction logs remain largely untapped as diagnostic signals. We present a pipeline that maps student questions from a conversational AI teaching assistant to curriculum topics using a few-shot text classifier, grounded in a GPT-4-extracted prerequisite knowledge graph of course concepts. Evaluated on 1,340 question events from 164 students in a graduate-level AI course, our classifier achieves 80.0% accuracy across 43 labels (42 curriculum topics plus an "unknown" abstention class). Topic-level question volume correlates significantly with student self-reported difficulty from an independent mid-semester survey (rho = 0.491, p = 0.008, n = 28 topics), providing convergent evidence that the classified question stream reflects genuine topic difficulty. These results demonstrate that conversational AI interaction logs, mapped onto curriculum structure, carry actionable signals about topic-level knowledge gaps and provide instructors with a curriculum-grounded view of which topics warrant attention.