Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a framework for generating large-scale instruction data for scientific diagrams by leveraging terminology from scientific curricula, addressing the challenges faced by vision-language models (VLMs) in understanding these complex visuals. The authors constructed the SciGram dataset, comprising over 194,000 diagrams and 1.4 million visual instructions, which significantly enhances the performance of models on diagram-centric benchmarks like TQA and ScienceQA. Fine-tuning on SciGram allows existing models, such as LLaVA OneVision, to achieve new state-of-the-art results in diagram question answering, demonstrating the potential of terminology-grounded instruction generation in improving VLMs' reasoning capabilities in scientific contexts.
Fine-tuning on the newly created SciGram dataset allows models to outperform state-of-the-art VLMs in understanding scientific diagrams, despite using fewer training instances.
Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images. However, they continue to struggle with scientific diagrams, which are designed to convey functional or relational meaning rather than literal scenes. We therefore introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula. Our approach systematically extracts domain concepts, synthesizes atomic facts, retrieves relevant diagrams from the web, and generates multimodal supervision in the form of diagram captions and multiple-choice questions. Using this pipeline, we construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Despite relying on noisy web data and synthetic annotations, models fine-tuned on SciGram achieve substantial improvements on diagram-centric benchmarks, including TQA, ScienceQA, and AI2D, outperforming or matching state-of-the-art VLMs while using fewer training instances. Furthermore, augmenting existing models such as LLaVA OneVision with SciGram establishes new state-of-the-art performance on diagram question answering. Our results highlight the effectiveness of terminology-grounded instruction generation as a general strategy for improving vision-language reasoning in scientific domains. To support future research in scientific diagram understanding, we release both the SciGram dataset and models.