Search papers, labs, and topics across Lattice.
This paper introduces MolSight, a graph-aware vision-language model that enhances the understanding of molecular images by integrating a Molecular Topology Module and a Molecular Grounding Module. By incorporating chemical-bond adjacency information and aligning visual features with chemical symbolic semantics, MolSight addresses the limitations of existing molecular vision-language models in structural alignment and topological modeling. Experimental results show that MolSight significantly outperforms current state-of-the-art models in various chemical visual understanding tasks, marking a substantial advancement in molecular image reasoning capabilities.
MolSight achieves unprecedented accuracy in molecular image reasoning by seamlessly integrating chemical topology with vision-language processing.
Using molecular large language models (LLMs) as a unified framework for understanding molecular structures and functions is emerging as a new trend in tasks such as molecular design and drug discovery. However, these models struggle to fully capture the visual representation of molecular structures, limiting their potential. While existing molecular vision-language models (VLMs) show promise, they still face challenges in structural alignment and lack the necessary topological modeling for accurate molecular understanding. To address this, we propose MolSight, a graph-aware vision-language model framework designed to enhance the understanding of molecular images by VLMs. MolSight integrates a Molecular Topology Module to inject chemical-bond adjacency information into vision tokens, and a Molecular Grounding Module to align visual features with chemical symbolic semantics. Our experiments demonstrate that MolSight significantly outperforms existing VLMs, molecular LLMs, and specialized tools across multiple chemical visual understanding tasks, achieving a new level of molecular image reasoning.
