Search papers, labs, and topics across Lattice.
This paper introduces a novel tutoring system that enables a large language model (LLM) to effectively withhold answers, thereby enhancing its educational value. Through a randomized study, it was found that a Socratically guarded version of the model maintained practice gains while eliminating subsequent test performance declines, highlighting the importance of answer-withholding in educational contexts. The system employs a multi-faceted architecture that includes a non-LLM policy core and a deterministic detector to enforce answer-withholding, achieving full compliance with established acceptance criteria through an automated tuning process.
Students using a Socratic LLM tutor retained knowledge better than those with an unguarded chatbot, revealing the critical role of answer-withholding in effective learning.
An effective large language model (LLM) tutor must often decline to give an answer it could easily produce. In a randomized study, students who used an unguarded chatbot scored higher while practicing but lower on a later test taken without it, whereas a Socratically guarded version of the same model kept the practice gain and removed the later loss [4]. Reliable answer-withholding is therefore central to a tutor's value, yet a capable model pressed by a frustrated student does not withhold reliably on a prompt alone. We report a deployed tutoring system that enforces answer-withholding as a per-turn, machine-checkable contract, and a method for tuning that withholding against evidence. A non-LLM policy core, reading only trusted learner state, sets a per-turn ceiling on an eight-rung help ladder; a deterministic detector strips solution code; and a separate LLM judge checks each risky reply against the contract. We tune the behavior with an automated evaluation that uses no human subjects: scripted student personas are driven through the live pipeline and re-scored by a stronger model, and we record each rejection's stated reason so failures are fixed by cause. Doing so revealed an interpretable"over-help ladder,"from blatant solution leaks, to naming the exact bug, to over-citing general facts, with each fix exposing the next. The tutor reached full compliance on all four acceptance criteria. We offer the measure, diagnose, and fix loop as a reusable recipe for any LLM agent that must refuse a capability it has.
