Search papers, labs, and topics across Lattice.
This paper investigates the practical application of a retrieval-augmented generation (RAG)-based LLM assistant in a university course, focusing on replicating a previously teacher-led exercise. Through a mixed-methods crossover design, the study assesses student motivation, perceptions of AI vs. human interaction, AI response quality, and the LLM's impact on academic performance. Results indicate the pedagogical feasibility of LLMs in specialized courses while highlighting challenges and opportunities for future integration.
Students perceive AI assistants as less intimidating and more approachable than human teachers, but also recognize limitations in specialized knowledge and nuanced feedback.
The landscape of education is changing rapidly, shaped by emerging pedagogical approaches, technological innovations such as artificial intelligence (AI), and evolving societal expectations, all of which demand thorough evaluation of new educational tools. Although large language models (LLMs) present substantial opportunities especially in Higher Education, their propensity to generate hallucinations and their limited specialized knowledge may introduce significant risks. This study aims to address these risks by examining the practical implementation of an LLM-enhanced assistant in a university level course. We implemented a generative AI assistant grounded in a retrieval-augmented generation (RAG) model to replicate a previously teacher-led, time-intensive exercise. To assess the effectiveness of the LLM, we conducted three separate experiments through iterative mixed-methods approaches, including a crossover design. The resulting data address central research questions related to student motivation, perceived differences between engaging with the LLM versus a human teacher, the quality of AI-generated responses, and the impact of the LLM on students'academic performance. The results offer direct insights into students'views and the pedagogical feasibility of embedding LLMs into specialized courses. Finally, we discuss the main challenges, opportunities and future directions of LLMs in teaching and learning in Higher Education.
