Search papers, labs, and topics across Lattice.
This study investigates the internal workings of Chemical Language Models (cLMs) by applying sparse autoencoders (SAEs) to MolFormer, revealing a layered approach to molecular representation. The findings indicate that early layers focus on syntactic parsing using position-tracking latents, while later layers capture more complex semantic features related to molecular structure and pharmacology. Notably, the research highlights that non-canonical SMILES lead to greater representation shifts than invalid SMILES, suggesting a nuanced understanding of molecular syntax and semantics within cLMs.
Non-canonical SMILES disrupt molecular representations more than invalid ones, revealing deeper insights into how cLMs understand molecular grammar and semantics.
Chemical language models (cLMs) are widely assumed to learn surface-level syntactic patterns rather than learning meaningful molecular semantics. Here, we apply sparse autoencoders (SAEs) to MolFormer, an encoder-only cLM, to mechanistically examine how molecular representations are built across layers. We discover that early layers rely on position-tracking latents to parse molecular grammar, while later layers encode atom-in-substructure and pharmacologically relevant features. Additionally, we show that non-canonical SMILES produce more disruptive representation shifts than invalid SMILES, driven by position-latent disruption propagating across layers. To support further exploration, we develop InterMol, an interactive visualizer for SAE activations on molecular strings and structures.
