Search papers, labs, and topics across Lattice.
This study explores the early detection of struggling students in CS1 programming by analyzing keystroke-level logs from the CodeBench platform. By comparing execution-based features with keystroke-level editing features, the authors find that the latter significantly enhances the prediction of students who are fully stuck, particularly at the earliest stages of coding exercises. The results indicate that behavioral signals captured at the start of an exercise can effectively identify students in need of timely support, highlighting the importance of keystroke-level data in educational interventions.
Keystroke-level logs reveal critical early indicators of student struggle, outperforming traditional execution logs in predicting coding success.
This paper investigates the feasibility of early detection of struggling students during CS1 programming exercises using keystroke-level logs. Some students fail to reach a correct solution before the exercise ends, and by the time this becomes apparent from grades or final outcomes, the opportunity for timely instructor support aimed at helping them recover may have passed. We use data from the CodeBench platform, which records real-time code editing events at the keystroke level, alongside execution and submission logs. We define two outcome groups: Breakthrough (BT) students, whose prior submissions all receive 0% and whose final submission achieves full credit, and Fully Stuck (FS) students, whose submissions all receive 0% without reaching a correct solution. To examine this feasibility, we focus on two questions: (RQ1) whether adding keystroke-level editing features improves the prediction of FS students over execution-log features alone, and (RQ2) at which stage BT and FS students can be predicted most accurately. Experiments on the 2019-1 semester of the CodeBench dataset, comprising 507 students, compare three feature configurations: execution-based features (ExecOnly), CodeMirror-based features (CMOnly), and their combination (Combined). We evaluate prediction across successive submission-based stages during each exercise. In the earliest stage, CMOnly outperforms ExecOnly (AUROC 0.654 vs. 0.575), and Combined further improves over ExecOnly by +0.098 (AUROC 0.674). Across all configurations, the earliest stage yielded the strongest predictive signal. These findings indicate that behavioral signals present at the very start of an exercise contain useful clues about whether a student will ultimately solve the problem, and that keystroke-level editing logs provide additional value for early prioritization beyond execution logs alone.
