Search papers, labs, and topics across Lattice.
This study investigates the interpretability of protein language model (PLM) embeddings by employing an orthogonal projection technique to isolate the effects of known biochemical features. The authors demonstrate that removing these features from the embeddings significantly degrades the performance of a classifier predicting protein fitness, indicating that the embeddings capture meaningful biochemical information. The findings reveal that PLM embeddings encode patterns related to biochemical properties, providing a quantifiable link between these embeddings and protein fitness predictions.
Orthogonal projection reveals that protein language model embeddings capture critical biochemical features, with their removal leading to significant drops in predictive performance.
Recently, there has been a growing adoption of protein language models (PLMs) in biomedical science. Their embeddings provide a rich numerical representation of protein sequences which achieve state-of-the-art performance on several downstream tasks including protein fitness prediction. However, PLM embeddings are not directly interpretable and, thereby, it remains unclear what features they encode. To gain insight into which biochemical properties of the protein are driving the prediction, we leverage an orthogonal projection technique that removes linear effects of known tabular features from embeddings and extend it to high-order and interaction effects. In this way, we remove the effects of interpretable biochemical features from PLM embeddings. In an ablation study, we show that this leads to a decrease in performance for a downstream classifier trained only on the embeddings to predict protein fitness. In an additional evaluation, we find that these biochemical features explain a substantial part of the variance in the predictions of this classifier. Hence, we can show that PLM embeddings encode patterns correlated with biochemical properties and quantify their contribution to predicting protein fitness. This computationally efficient approach is not limited to the features or embeddings considered here and is readily transferable to problem settings beyond protein fitness prediction.
