Search papers, labs, and topics across Lattice.
This paper investigates how well LLMs can model student misconceptions when generating multiple-choice distractors, a task requiring solution knowledge, misconception simulation, and plausibility evaluation. They introduce a taxonomy to analyze LLM strategies, comparing them to learning science best practices. The analysis reveals that LLMs generally solve the problem correctly first, then simulate misconceptions, and finally select distractors, with errors primarily stemming from failures in finding the correct solution or selecting among candidates.
LLMs surprisingly mimic human strategies for generating plausible student misconceptions, but their success hinges on first solving the problem correctly.
Modeling plausible student misconceptions is critical for AI in education. In this work, we examine how large language models (LLMs) reason about misconceptions when generating multiple-choice distractors, a task that requires modeling incorrect yet plausible answers by coordinating solution knowledge, simulating student misconceptions, and evaluating plausibility. We introduce a taxonomy for analyzing the strategies used by state-of-the-art LLMs, examining their reasoning procedures and comparing them to established best practices in the learning sciences. Our structured analysis reveals a surprising alignment between their processes and best practices: the models typically solve the problem correctly first, then articulate and simulate multiple potential misconceptions, and finally select a set of distractors. An analysis of failure modes reveals that errors arise primarily from failures in recovering the correct solution and selecting among response candidates, rather than simulating errors or structuring the process. Consistent with these results, we find that providing the correct solution in the prompt improves alignment with human-authored distractors by 8%, highlighting the critical role of anchoring to the correct solution when generating plausible incorrect student reasoning. Overall, our analysis offers a structured and interpretable lens into LLMs' ability to model incorrect student reasoning and produce high-quality distractors.
