Search papers, labs, and topics across Lattice.
VisBrowse-Bench, a new benchmark, is introduced to evaluate visual reasoning in multimodal browsing agents by focusing on native visual information within web pages. The benchmark contains 169 VQA instances across diverse domains and uses multimodal evidence cross-validation via text-image retrieval and joint reasoning to assess visual reasoning during search. Experiments using this benchmark reveal that even state-of-the-art models like Claude-4.6-Opus and o3-deep-research achieve accuracies of only 47.6% and 41.1% respectively, highlighting the limitations of current MLLMs in visual-native search.
Current multimodal browsing agents are surprisingly bad at using visual information on webpages, with even top models scoring below 50% accuracy on a new visual-native search benchmark.
The rapid advancement of Multimodal Large Language Models (MLLMs) has enabled browsing agents to acquire and reason over multimodal information in the real world. But existing benchmarks suffer from two limitations: insufficient evaluation of visual reasoning ability and the neglect of native visual information of web pages in the reasoning chains. To address these challenges, we introduce a new benchmark for visual-native search, VisBrowse-Bench. It contains 169 VQA instances covering multiple domains and evaluates the models' visual reasoning capabilities during the search process through multimodal evidence cross-validation via text-image retrieval and joint reasoning. These data were constructed by human experts using a multi-stage pipeline and underwent rigorous manual verification. We additionally propose an agent workflow that can effectively drive the browsing agent to actively collect and reason over visual information during the search process. We comprehensively evaluated both open-source and closed-source models in this workflow. Experimental results show that even the best-performing model, Claude-4.6-Opus only achieves an accuracy of 47.6%, while the proprietary Deep Research model, o3-deep-research only achieves an accuracy of 41.1%. The code and data can be accessed at: https://github.com/ZhengboZhang/VisBrowse-Bench
