Search papers, labs, and topics across Lattice.
The authors evaluated 40 multimodal LLMs, including proprietary models like GPT-5 and Gemini-2.5-Pro, on a new benchmark of US National Chemistry Olympiad questions requiring integrated visual and textual reasoning. They found that many models struggle with modality fusion, with performance sometimes decreasing when images are included, highlighting a misalignment in vision-language integration. Chain-of-Thought prompting was shown to improve both accuracy and visual grounding, as demonstrated through ablation studies and occlusion-based interpretability.
Multimodal LLMs sometimes perform *worse* on chemistry problems when given images, revealing surprising failures in vision-language fusion.
Multimodal scientific reasoning remains a significant challenge for large language models (LLMs), particularly in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, we systematically evaluate 40 proprietary and open-source multimodal LLMs, including GPT-5, o3, Gemini-2.5-Pro, and Qwen2.5-VL, on a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. These questions require integrated visual and textual reasoning across diverse modalities. We find that many models struggle with modality fusion, where, in some cases, removing the image even improves accuracy, indicating misalignment in vision-language integration. Chain-of-Thought prompting consistently enhances both accuracy and visual grounding, as demonstrated through ablation studies and occlusion-based interpretability. Our results reveal critical limitations in the scientific reasoning abilities of current MLLMs, providing actionable strategies for developing more robust and interpretable multimodal systems in chemistry. This work provides a timely benchmark for measuring progress in domain-specific multimodal AI and underscores the need for further advances at the intersection of artificial intelligence and scientific reasoning. Multimodal scientific reasoning remains a significant challenge for large language models in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, the authors systematically evaluate 40 proprietary and open-source multimodal large language models on a curated benchmark of Olympiad-style chemistry questions, revealing that some models struggle with vision-language fusion, while chain-of-thought prompting can enhance both accuracy and visual grounding.
