Search papers, labs, and topics across Lattice.
This study systematically probes chemical language models (CLMs) to understand the molecular substructures they encode, analyzing 78 substructures across eight pre-trained and six randomly initialized models. The findings reveal that pre-training enhances the models' awareness of molecular structures, especially in the upper layers, while randomly initialized models demonstrate a strong ability to encode ring structures from the outset. Additionally, fine-tuning on chemical tasks significantly alters representations of task-relevant substructures, aligning with established chemical theory.
Pre-training boosts CLMs' molecular structure awareness, but surprisingly, even randomly initialized models excel at encoding ring structures from the first layer.
Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster a better understanding of CLMs, we conduct a systematic study and probe for 78 molecular substructures across eight pre-trained and six randomly initialized models. We furthermore study how fine-tuning on chemical downstream tasks affects the learned representations of molecular substructures. Our results show that pre-training generally improves molecular structure awareness of CLMs, particularly in the upper layers. Moreover, randomly initialized models already encode ring structures well in the first layer. Our analysis on two chemical downstream tasks further reveals that, interestingly, fine-tuning affects task-relevant molecular substructures more than others, indicating that the changes in the representations follow chemical theory.
