Search papers, labs, and topics across Lattice.
MarkushGrapher-2 is introduced as an end-to-end approach for recognizing chemical structures in documents by jointly encoding text, image, and layout information using a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder. To train the model, the authors created a large-scale dataset of real-world Markush structures using an automatic pipeline and a manually-annotated benchmark, IP5-M. Experiments demonstrate that MarkushGrapher-2 significantly outperforms existing models in multimodal Markush structure recognition while maintaining strong performance in molecule structure recognition.
End-to-end recognition of complex chemical structures from documents is now possible, thanks to a new model and dataset that leapfrog existing methods.
Automatically extracting chemical structures from documents is essential for the large-scale analysis of the literature in chemistry. Automatic pipelines have been developed to recognize molecules represented either in figures or in text independently. However, methods for recognizing chemical structures from multimodal descriptions (Markush structures) lag behind in precision and cannot be used for automatic large-scale processing. In this work, we present MarkushGrapher-2, an end-to-end approach for the multimodal recognition of chemical structures in documents. First, our method employs a dedicated OCR model to extract text from chemical images. Second, the text, image, and layout information are jointly encoded through a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder. Finally, the resulting encodings are effectively fused through a two-stage training strategy and used to auto-regressively generate a representation of the Markush structure. To address the lack of training data, we introduce an automatic pipeline for constructing a large-scale dataset of real-world Markush structures. In addition, we present IP5-M, a large manually-annotated benchmark of real-world Markush structures, designed to advance research on this challenging task. Extensive experiments show that our approach substantially outperforms state-of-the-art models in multimodal Markush structure recognition, while maintaining strong performance in molecule structure recognition. Code, models, and datasets are released publicly.
