Search papers, labs, and topics across Lattice.
This paper introduces PyMETA, a comprehensive dataset for hierarchical classification of code errors in student submissions, comprising 48,646 samples with single-error labels and a diagnostic subset of 97 multi-error samples. The dataset employs a three-level hierarchical taxonomy based on Python's official exception hierarchy, facilitating nuanced error analysis. Evaluation of multiple models reveals that while Gemini 2.5 Pro achieves an 81.8% macro F1 score for multi-error prompting, finetuned smaller models outperform prompted LLMs, highlighting significant disparities in classification performance across error types.
Prompted LLMs struggle with code error classification, often misclassifying logic errors, while smaller finetuned models lead the way in accuracy.
With the advancement of Large Language Models (LLMs), code error detection has extended beyond traditional IDE diagnostics to context-sensitive debugging in educational scenarios. However, existing approaches lack large-scale datasets, multi-error analysis, and unified error taxonomies. To address this, we introduce PyMETA, a large-scale Python code error classification dataset of 48,646 student submissions, with single-error labels for all samples and a diagnostic subset of 97 expert-annotated multi-error samples. The dataset uses a three-level hierarchical taxonomy, from a binary error/no-error split down to 14 fine-grained error types grounded in Python's official exception hierarchy. We evaluate multi-level classification tasks on two finetuned models and four LLMs with prompting, comparing their classification performance and runtime cost. For multi-error prompting, the best model, Gemini 2.5 Pro, achieves 81.8% macro F1 under the"contains"criterion. We observe that: 1) prompted LLMs still underperform finetuned smaller models; 2) models exhibit significant disparities across error types; 3) most LLMs over-classify code as Logic Error, with GPT-3.5 showing the highest Logic Error Overprediction Rate and Gemini 2.5 Pro the lowest. Our work establishes a data foundation and provides insights for LLM-based code error research.
