Search papers, labs, and topics across Lattice.
This paper introduces Diagram-MMU, a multi-modal benchmark designed to evaluate the capabilities of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark comprises 3.7k curated diagrams and 18.3k human-validated questions across six domains, focusing on tasks such as diagram-to-code parsing, editing, and question answering. Results show that while MLLMs perform well in reasoning over diagrams, they struggle significantly with diagram-to-code tasks, highlighting a critical gap in their capabilities and the need for improved methods in this area.
MLLMs excel at reasoning over diagrams but falter in parsing and editing them, revealing a significant gap that needs addressing.
Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.
