Search papers, labs, and topics across Lattice.
The paper introduces ScratchMath, a new benchmark dataset of 1,720 handwritten math scratchwork samples from Chinese students, designed for error cause explanation (ECE) and error cause classification (ECC). They evaluated 16 MLLMs on ScratchMath, finding significant performance gaps compared to human experts, particularly in visual recognition and logical reasoning within the handwritten context. The results highlight the limitations of current MLLMs in understanding and diagnosing errors in complex, multimodal educational data.
Despite their visual reasoning prowess, today's MLLMs still struggle to understand handwritten math scratchwork, falling far short of human expert performance in diagnosing student errors.
Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an"examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.
