Search papers, labs, and topics across Lattice.
This paper introduces MatMMExtract, an innovative pipeline that disaggregates compound figures from materials science literature into individual sub-panels, enabling structured annotations through a large language model aligned with a curated taxonomy. By applying this method to 14,810 articles, the authors generated MatSciFig, a dataset comprising 391,606 image-text pairs, significantly enhancing the accessibility of visual data in the field. The results show that a fine-tuned YOLO12-m detector achieves high accuracy in panel localization, while the best-performing language model, Gemini 3.1 Flash Lite, provides a favorable balance of quality and cost in annotation generation, leading to a substantial improvement in retrieval performance.
Unlocking the visual data in materials science literature could revolutionize how researchers access and utilize experimental knowledge encoded in figures.
The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.
