Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces EduClaw-Bench, a novel benchmark designed to evaluate pedagogical LLM agents over a long-term interaction with simulated learners, reflecting the sustained nature of real-world tutoring. By grounding the evaluation in knowledge tracing and assessing agents across 55 scenarios, the study reveals that effective tutoring depends on the synergy between the base model and the agent rather than on either in isolation. Notably, the findings indicate that few agent configurations can maintain high-quality tutoring throughout a 30-day period, underscoring the challenges in developing reliable educational AI systems.
Effective tutoring hinges on the interplay between LLM base models and agent configurations, revealing that most combinations fail to sustain quality over time.
Large language models (LLMs) power educational applications from tutoring to essay scoring, but each is a point solution to a single task, and only recently have these point solutions been integrated into agents operating over a learning management system (LMS). Yet tutoring is long-horizon, since a learner improves over days and weeks rather than in a single turn, and no benchmark evaluates an agent tutor across a sustained relationship. We introduce EduClaw-Bench, a benchmark that places an agent tutor in a continuous 30-day relationship with a simulated learner grounded in knowledge tracing (KT), whose knowledge-concept mastery, from a KT model trained on real-student data, drives its answers and is probed for learning gain across 55 scenarios. Each agent is scored on three primary axes (learning gain, responsiveness, and helpfulness) and two curriculum-design axes (Gagn\'e and Rosenshine), with helpfulness and the curriculum axes judged by a cross-family panel of three LLM judges. Evaluating 10 agent adapters over three base-model tiers yields two findings that single-tier, single-session evaluation cannot reach. First, tutoring quality belongs to the base model and the agent harness together rather than either alone. Second, almost no combination sustains good tutoring over the full horizon. A calibration check ($\text{ECE}=0.049$) and a live-classroom field study confirm that the simulated learner and its measurements track reality. Our work is a step toward trustworthy AI tutors for future education.