Search papers, labs, and topics across Lattice.
Raven, a novel automated assessment framework for Scratch programs, addresses the limitations of traditional assertion-based grading by using instructor-specified video generation rules shared across student submissions. It leverages LLMs and video analysis to evaluate program behavior against grading criteria, eliminating the need for explicit test cases. Experiments on 13 assignments with 140 student submissions demonstrate that Raven significantly outperforms existing tools in grading accuracy and robustness, validated by a classroom study.
Grading Scratch programs just got easier: Raven uses LLMs to watch student-created videos and automatically assess whether they meet the assignment's goals.
Block-based programming environments such as Scratch are widely used in introductory computing education, yet scalable and reliable automated assessment remains elusive. Scratch programs are highly heterogeneous, event-driven, and visually grounded, which makes traditional assertion-based or test-based grading brittle and difficult to scale. As a result, assessment in real Scratch classrooms still relies heavily on manual inspection and delayed feedback, introducing inconsistency across instructors and limiting scalability. We present Raven, an automated assessment framework for Scratch that replaces program-specific state assertions with instructor-specified, task-level video generation rules shared across all student submissions. Raven integrates large language models with video analysis to evaluate whether a program's observed visual and interactive behaviors satisfy grading criteria, without requiring explicit test cases or predefined outputs. This design enables consistent evaluation despite substantial diversity in implementation strategies and interaction sequences. We evaluate Raven on 13 real Scratch assignments comprising over 140 student submissions with ground-truth labels from human graders. The results show that Raven significantly outperforms prior automated assessment tools in both grading accuracy and robustness across diverse programming styles. A classroom study with 30 students and 10 instructors further demonstrates strong user acceptance and practical applicability. Together, these findings highlight the effectiveness of task-level behavioral abstractions for scalable assessment of open-ended, event-driven programs.
