Search papers, labs, and topics across Lattice.
This study benchmarked the performance of 11 AI tools (6 general-purpose and 5 education-specific) in classifying the cognitive demand of mathematical tasks across four levels. The AI tools achieved only 63% average accuracy, with no tool exceeding 83%, and struggled particularly with tasks at the extremes of cognitive demand. Error analysis revealed that the tools overweighted surface textual features and exhibited weaknesses in reasoning about factors influencing cognitive demand, suggesting a bias towards middle-category levels.
AI tools are surprisingly bad at classifying the cognitive demand of math problems, with accuracy barely above chance and a systematic bias towards average difficulty, raising concerns about their utility in supporting teachers.
Teachers face increasing demands on their time, particularly in adapting mathematics curricula to meet individual student needs while maintaining cognitive rigor. This study evaluates whether AI tools can accurately classify the cognitive demand of mathematical tasks, which is important for creating or adapting tasks that support student learning. We tested eleven AI tools: six general-purpose (ChatGPT, Claude, DeepSeek, Gemini, Grok, Perplexity) and five education-specific (Brisk, Coteach AI, Khanmigo, Magic School, School.AI), on their ability to categorize mathematics tasks across four levels of cognitive demand using a research-based framework. The goal was to approximate the performance teachers will achieve with straightforward prompts. On average, AI tools accurately classified cognitive demand in only 63% of cases. Education-specific tools were not more accurate than general-purpose tools, and no tool exceeded 83% accuracy. All tools struggled with tasks at the extremes of cognitive demand (Memorization and Doing Mathematics), exhibiting a systematic bias toward middle-category levels (Procedures with/without Connections). The tools often gave plausible-sounding explanations likely to be persuasive to novice teachers. Error analysis of AI tools' misclassification of the broad level of cognitive demand (high vs. low) revealed that tools consistently overweighted surface textual features over underlying cognitive processes. Further, AI tools showed weaknesses in reasoning about factors that make tasks higher vs. lower cognitive demand. Errors stemmed not from ignoring relevant dimensions, but from incorrectly reasoning about multiple task aspects. These findings carry implications for AI integration into teacher planning workflows and highlight the need for improved prompt engineering and tool development for educational applications.
CMU ML