Search papers, labs, and topics across Lattice.
This study analyzes 13 protein language models (PLMs) across 15 downstream tasks (DTs) from 11 datasets to understand the informativeness of embeddings from intermediate layers. The researchers trained probe models on embeddings from various layers and found that the last layers of PLMs often do not yield the best performance for DTs, with embeddings from shallower layers outperforming in certain contexts, particularly with deep mutational scan datasets. A key insight is that the distribution of relevant information across PLM layers is closely tied to the nature of the dataset rather than the task itself, revealing a nuanced relationship between task objectives and embedding utility.
Last-layer embeddings in protein language models often underperform, with shallow layers proving more effective for specific datasets like deep mutational scans.
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By a common consensus, embeddings from the model's last layer are used, and the model's internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs from 11 datasets to investigate the informativeness of embeddings created in intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and computed characteristics of the latent spaces they span to estimate the information they contain, and found that the last layers of PLMs rarely contained embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between DTs and the distribution across PLMs'layers of the relevant information to predict that task. For example, similarity between the pre-training objective and the objective of predicting properties of individual residues leads to a steady increase in understanding of such tasks across the layers of PLMs. On the other hand, for whole-protein tasks, we observe that the dataset, rather than the task itself, defines PLMs'ability to perform well on a DT. Embeddings from shallow layers of PLMs perform better for datasets that contain deep mutational scan (DMS) data, while datasets containing diverse natural proteins find most useful embeddings in the models'deeper layers. Additionally, we discover that the performance of PLMs drops significantly when tasks are introduced for artificial proteins.
