Search papers, labs, and topics across Lattice.
This paper introduces a multimodal chatbot designed for university stakeholders, leveraging a retrieval-augmented generation (RAG) approach to provide accurate, context-based responses to complex queries. By integrating a large language model with semantic retrieval and enabling both text and image inputs, the system addresses the limitations of traditional rule-based chatbots, particularly in developing countries. Key results indicate a significant reduction in response hallucination from 31.7% to 6.6%, demonstrating the effectiveness of retrieval grounding in enhancing information reliability and user satisfaction.
Hallucination rates plummet from 31.7% to 6.6% in a new multimodal chatbot that adapts to university stakeholders' complex queries.
University stakeholders often face difficulties in accessing timely and reliable information, especially in developing countries, where there are very few intelligent support systems. Existing rule-based chatbots are unable to handle complex, domain-specific queries and are not well-equipped to adapt to evolving institutional policies. As a fill-in-the-gap solution, we present the multimodal university chatbot with retrieval-augmented generation. The system combines the large language model with semantic retrieval to produce context-based responses from institution-centric resources, such as the university handbook. The system accepts text and image queries through the vision-language model and applies quantized inference for rapid deployment on constrained hardware. A scalable backend built with FastAPI, adjoined with a responsive frontend developed with Next.js, ensures real-time usability. Our multimodal evaluation demonstrates that the system maintains strong satisfaction scores across both text and image queries, despite increased response time for visual inputs. Furthermore, quantitative evaluation shows that hallucination is reduced from 31.7% to 6.6% in our proposed RAG-based system, confirming the effectiveness of retrieval grounding.
