Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of different types of AI-generated feedback in an introductory Python programming course through a randomized classroom experiment involving 215 students and 6,693 submissions. The findings reveal that natural language hints significantly improve student completion rates and accelerate the learning process, while the impact of test case feedback varies based on its validity. The research emphasizes the importance of assessing feedback quality to optimize educational outcomes in programming instruction.
Natural language feedback boosts student success in programming courses, while test case feedback's impact hinges on its validity.
Large language models (LLMs) are increasingly used to provide automated feedback in introductory programming courses, yet empirical evidence from authentic classroom deployments comparing different feedback modalities remains limited. In this work, we present a large-scale classroom study in which AI-generated feedback was deployed through a randomized protocol in an introductory Python programming course. Students received one of three feedback conditions on incorrect submissions: natural language hints, AI-generated failing test cases, or no AI feedback. We release the resulting dataset, ProgFeed, which captures 6,693 submissions from 215 consenting students across 17 labs, including feedback conditions, execution-based performance measures, and fine-grained temporal information. Using this data, we analyze learning trajectories, feedback quality, and submission behavior over repeated attempts. We find that natural language feedback is significantly associated with higher completion rates and faster convergence to correct solutions. Test case feedback, by contrast, exhibits heterogeneous effects that depend critically on feedback validity. Our results suggest that the form of AI-generated feedback matters, and that evaluating feedback quality -- not just its presence -- is essential for understanding its pedagogical impact.