Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This study systematically evaluates the downstream performance of chemical language models (CLMs) pretrained on large molecular corpora across 64 property prediction tasks, revealing that pretraining gains saturate and fall below gradient-boosting baselines. Only supervised pretraining on physico-chemical descriptors consistently improved performance, with some pretraining strategies yielding worse results than random initialization. The findings suggest CLMs primarily learn descriptor-like properties and token statistics, with limited emergent chemistry beyond classical molecular features.
Pretraining chemical language models yields diminishing returns, often underperforming classical baselines and even random initialization, suggesting current approaches capture little true chemistry.
Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.