Search papers, labs, and topics across Lattice.
This paper introduces CodeInsight, a dataset comprising over 3 million submissions from undergraduate students in C++ courses, aimed at modeling iterative problem-solving through feedback and revision. By employing a benchmark that includes a Recurrent State Space Model (RSSM) and an LLM-based predictor, the study reveals that while the RSSM excels in predictive accuracy, the LLM's generative capabilities provide valuable insights into failure modes despite lower accuracy. Notably, the research uncovers an inverse relationship between coding proficiency and predictive performance, suggesting that LLMs function better as context-conditioned generative solvers rather than reliable predictors of solver behavior.
Iterative problem-solving reveals that LLMs, while less accurate, can generate insightful solutions that illuminate failure modes in student coding attempts.
Solving problems through repeated attempts is a sequential modeling task: at each step, the solver receives feedback and decides how to revise their solutions. Predicting whether performance improves, plateaus, or regresses across attempts is central to understanding any iterative problem-solving process in both human learners and autonomous agents. Beyond outcomes, modeling what errors persist and how strategies shift across attempts provides deeper insight into the mechanics of sequential learning. Studying these dynamics requires observing many solvers as they attempt, receive feedback, and revise. Programming courses with automated grading provide this setting, as students iteratively submit code to test suites and receive feedback on every attempt. We therefore curate CodeInsight, a large-scale dataset of over 3 million submissions from 3,286 undergraduates across 2 introductory C++ courses in 2 academic years, with test-case-level outcomes, timestamps, and source code. On this dataset, we build a benchmark that evaluates models spanning parametric, sequential, and generative traditions under a shared calibration-and-scoring protocol, including a Recurrent State Space Model (RSSM) adapted to track solver characteristics through discrete latent variables and an LLM-based predictor that generates explicit solutions. The adapted RSSM achieves the strongest predictive accuracy on three of the four courses. The LLM predictor is less accurate but produces full submissions at each attempt, enabling direct analysis of failure modes. We find that the model's coding proficiency is inversely related to predictive performance in this setting, with the LLM better understood as a generative solver conditioned on context rather than a faithful predictor of solver behavior. We publicly release our code and the dataset on request to facilitate future research.
NUS
Stanford HAI