Search papers, labs, and topics across Lattice.
This paper investigates the representational differences between Protein Language Models (PLMs) and Natural Language Models (NLMs) by comparing the distribution of information across attention head layers. It finds that PLMs and NLMs store information differently across layers, likely due to the distinct characteristics of protein and natural language. The authors then adapt an early-exit technique to PLMs, enabling task-specific selection of intermediate layer representations, which improves both accuracy (0.4-7.01%) and efficiency (>10%) in non-structural protein property prediction.
Protein language models, despite borrowing NLP architectures, distribute information across layers very differently, but a simple early-exit strategy can exploit this to boost both accuracy and efficiency.
Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.
CMU ML