Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the pedagogical behavior of large language models (LLMs) in tutoring contexts by developing a validated five-level scaffolding scale to assess the degree of assistance provided in student interactions. Analyzing 14,637 responses from 203 students in an AI course, the research reveals that over 95% of LLM responses offer high levels of assistance, primarily categorized as Explaining or Solving. While scaffolding level influences students' conversational behavior, it does not significantly predict exam performance beyond prior achievement and dialogue behavior, establishing a baseline for evaluating LLM tutoring effectiveness.
Over 95% of LLM tutoring responses provide high levels of assistance, raising questions about the effectiveness of such support in student learning outcomes.
Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.