Search papers, labs, and topics across Lattice.
This paper introduces CheMatE, a bi-semantic embedding model that integrates molecular structure and domain-specific natural language into a unified representation space. Utilizing a two-stage training approach that combines masked language modeling with contrastive learning, CheMatE leverages a large-scale corpus of SMILES-annotated scientific documents to enhance its understanding of both chemical syntax and semantics. The model demonstrates strong performance in molecular property prediction and scientific language understanding, outperforming existing specialized chemistry models and general-purpose language models.
CheMatE achieves competitive performance by seamlessly integrating chemical structure with natural language understanding, challenging the notion that domain-specific models must sacrifice generality.
Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry. However, domain-adaptive pre-training often causes models to overfit to chemical syntax, catastrophically forgetting their foundational semantic capabilities. To address this challenge, we introduce CheMatE, a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language within the same representation space. Built on a ModernBERT backbone, CheMatE learns bi-semantic representations through a two-stage training procedure: continued masked language modeling (MLM) followed by a Matryoshka contrastive learning stage via Multiple Negative Ranking Loss (MNRL). First, we train the model using MLM on a novel, large-scale corpus of SMILES-annotated, long-context scientific documents that were constructed and curated from FineWeb and ChemPile (comprising 10.4B and 11.5B tokens, respectively). Subsequently, the model undergoes contrastive learning using a synthetic dataset of SMILES-text pairs algorithmically derived from our original training corpus. This design exposes the model to SMILES-enriched scientific literature, enabling bi-semantic understanding. We evaluate CheMatE across a range of downstream tasks covering molecular property prediction and scientific language understanding. Our results demonstrate that coupling our custom-curated datasets with this sequential training strategy yields robust, highly transferable representations. By effectively unifying structural and contextual signals within a single text-based framework, CheMatE achieves competitive performance across both specialized chemistry models and general-purpose language model baselines.
