Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of multimodal conversational AI on learning outcomes in visually-rich STEM domains by comparing a multimodal document-grounded conversational AI (MuDoC), a text-only conversational AI (TexDoC), and a textbook interface (DocSearch) in a randomized controlled study (N=124). MuDoC, which interleaved text and images, led to the highest post-test scores and most positive learning experience, while TexDoC, despite being more engaging than DocSearch, resulted in the lowest scores. The findings suggest that conversationality reduces extraneous cognitive load, while multimodality increases germane load, jointly optimizing learning outcomes.
Conversational AI can *hurt* learning in visually-rich domains if it's not paired with multimodal content, revealing a surprising disconnect between perceived engagement and actual knowledge gained.
Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content. However, while conversational AI is known to boost engagement, its impact on learning in visually-rich STEM domains remains under-explored. Moreover, there is limited understanding of how multimodality and conversationality jointly influence learning in generative AI systems. This work reports findings from a randomized controlled online study (N = 124) comparing three approaches to learning biology from textbook content: (1) a document-grounded conversational AI with interleaved text-and-image responses (MuDoC), (2) a document-grounded conversational AI with text-only responses (TexDoC), and (3) a textbook interface with semantic search and highlighting (DocSearch). Learners using MuDoC achieved the highest post-test scores and reported the most positive learning experience. Notably, while TexDoC was rated as significantly more engaging and easier to use than DocSearch, it led to the lowest post-test scores, revealing a disconnect between student perceptions and learning outcomes. Interpreted through the lens of the Cognitive Load Theory, these findings suggest that conversationality reduces extraneous load, while visual-verbal integration induced by multimodality increases germane load, leading to better learning outcomes. When conversationality is not complemented by multimodality, reduced cognitive effort may instead inflate perceived understanding without improving learning outcomes.