Search papers, labs, and topics across Lattice.
This paper introduces NAILA, an LLM-powered tool that provides automated feedback on student software engineering exercises, addressing challenges of scale and GenAI influence in introductory SE courses. The study evaluates NAILA's adoption, user acceptance, engagement, and impact on academic performance through an empirical study with over 900 students. Results show that students perceive NAILA as useful and easy to use, with engagement correlating with improved academic performance compared to traditional human feedback.
LLM-generated feedback can improve student performance in introductory software engineering courses, potentially surpassing traditional human feedback at scale.
Introductory Software Engineering (SE) courses face rapidly increasing student enrollment numbers, participants with diverse backgrounds and the influence of Generative AI (GenAI) solutions. High teacher-to-student ratios often challenge providing timely, high-quality, and personalized feedback a significant challenge for educators. To address these challenges, we introduce NAILA, a tool that provides 24/7 autonomous feedback for student exercises. Utilizing GenAI in the form of modern LLMs, NAILA processes student solutions provided in open document formats, evaluating them against teacher-defined model solutions through specialized prompt templates. We conducted an empirical study involving 900+ active students at the University of Duisburg-Essen to assess four main research questions investigating (1) the underlying motivations that drive students to either adopt or reject NAILA, (2) user acceptance by measuring perceived usefulness and ease of use alongside subjective learning progress, (3) how often and how consistently students engage with NAILA, and (4) how using NAILA to receive AI feedback impacts on academic performance compared to human feedback.
