Search papers, labs, and topics across Lattice.
This paper introduces SciMIF, a benchmark designed to evaluate Multimodal Large Language Models (MLLMs) in their ability to follow complex scientific instructions across 22 tasks in five scientific disciplines. The authors develop a comprehensive taxonomy of 10 constraint groups to systematically assess MLLMs, revealing significant performance disparities, particularly in chemistry, where models struggle with fine-grained constraints and deep disciplinary knowledge. The findings indicate that simply increasing model scale does not improve adherence to these constraints, highlighting critical limitations in current MLLMs for scientific applications.
MLLMs exhibit significant weaknesses in following scientific instructions, especially in chemistry, where they fail to meet fine-grained constraints despite increased model size.
Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields. In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions. Specifically, based on an extensive analysis of 22 distinct tasks across 5 representative scientific disciplines, we propose a comprehensive taxonomy comprising 10 constraint groups that captures both general functional requirements and discipline-specific characteristics. Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets. We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs. Our findings reveal significant performance disparities across different scientific disciplines, with chemistry posing greater challenges for current MLLMs. Furthermore, we observe that increasing the model scale does not yield corresponding improvements in constraint adherence, and current models still struggle severely with fine-grained constraints and instructions requiring the deep application of disciplinary knowledge. SciMIF fills the current void in evaluating multimodal instruction adherence within scientific domains, laying a crucial foundation for future enhancements of MLLMs in rigorous scientific applications. Data and code will be released at https://github.com/shenye7436/SciMIF .
