Search papers, labs, and topics across Lattice.
This study investigates the use of large language models (LLMs) to curate targeted video segments for answering student inquiries in introductory programming courses, aiming to reduce risks associated with generative AI such as hallucinations. The research benchmarks two proprietary models and one open-weight model against human-selected video content, revealing that the proprietary models can deliver answers that are both relevant and sufficient, despite not aligning perfectly with human selections. A pilot deployment in a large C programming cohort indicated strong student engagement, suggesting that this method can effectively integrate LLMs into educational settings while maintaining content reliability.
Proprietary LLMs can match human educators in delivering relevant answers from curated lecture videos, potentially transforming how programming education leverages AI.
This study evaluates the effectiveness of utilising large language models (LLMs) to retrieve targeted segments from delivered video recordings to answer student questions in introductory programming environments. By restricting AI to identifying existing, educator-verified media rather than generating open-ended text, this approach aims to mitigate common pedagogical risks such as generative hallucinations and cognitive bypassing. We benchmarked three distinct models, two proprietary (Gemini 3.1 Pro and GPT 5.4 Pro) and one open-weight (Qwen3.5 397B), against a human lecturer's manual video selections. An automated judging framework subsequently assessed the outputs for relevance, sufficiency, redundancy, and the presence of extraneous material. While the AI-retrieved timestamps rarely shared exact overlaps with the human baseline, the proprietary models achieved near-parity with the expert in delivering sufficient and highly relevant answers. Furthermore, a pilot deployment of this retrieval system in a large C programming cohort (n~=900) demonstrated strong user engagement, with students primarily utilising the tool to review foundational concepts. By leveraging AI to retrieve established lecture material, this approach shows potential for a reliable, high-fidelity pathway for safely integrating LLMs into novice computing courses.
