Search papers, labs, and topics across Lattice.
This paper introduces VIABench, a novel video benchmark tailored for evaluating Multimodal Large Language Models (MLLMs) in the context of assisting visually impaired individuals (VIIs). The benchmark encompasses three core tasks鈥擯roactive Reminder, Visual Question Answering (VQA), and Vision-Guided Interaction鈥攄esigned to assess the models' capabilities in real-world scenarios. Experimental results reveal that current MLLMs struggle significantly, particularly in the Proactive Reminder task, highlighting the need for further research and development in this area to enhance navigation and interaction for VIIs.
Current MLLMs fail to provide adequate support for visually impaired individuals, particularly in anticipating navigation-critical events in real-time.
Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achieved impressive results on general vision and language tasks, their practical utility in real-world blind assistance still remains largely underexplored. To fill this gap, we introduce VIABench, a comprehensive video benchmark specifically designed to evaluate MLLMs in Visually Impaired Assistance scenarios using first-person videos recorded or shared by VIIs themselves. VIABench defines three core tasks, each targeting a distinct requirement in visual assistance. Proactive Reminder: Assesses the model's ability to interpret ongoing video content while proactively anticipating and verbally describing upcoming navigation-critical events; Visual Question Answering (VQA): Evaluates the model's capacity to answer user-posed questions about the environment or objects within the video; Vision-Guided Interaction: Tests context-aware reasoning to accomplish intentional interactions between user and environment. To ensure a robust and fair evaluation, we propose a rigorous benchmarking pipeline that supports both online (real-time) and offline settings. Our experiments demonstrate that current MLLMs still struggle to deliver comprehensive support for VIIs, especially in the Proactive Reminder task, which demands accurate anticipation and real-time responsiveness. We hope VIABench will drive future research toward developing customized MLLMs for real-world assistance, ultimately improving navigation and interaction experiences for visually impaired individuals. Code and data will be released at https://github.com/MCG-NJU/VIABench.
