Search papers, labs, and topics across Lattice.
This paper investigates how protein language models (PLMs) detect exact and approximate sequence repeats, which are crucial for protein structure and function. They find that PLMs use a two-stage mechanism: first, building feature representations with general positional attention and biologically specialized components (neurons encoding amino-acid similarity), and second, using induction heads to attend to aligned tokens across repeats. The study demonstrates how PLMs combine language-based pattern matching with biological knowledge to solve this biological task.
Protein language models detect approximate sequence repeats using a two-stage mechanism where induction heads attend to aligned tokens, functionally subsuming the detection of exact repeats.
Protein sequences are abundant in repeating segments, both as exact copies and as approximate segments with mutations. These repeats are important for protein structure and function, motivating decades of algorithmic work on repeat identification. Recent work has shown that protein language models (PLMs) identify repeats, by examining their behavior in masked-token prediction. To elucidate their internal mechanisms, we investigate how PLMs detect both exact and approximate repeats. We find that the mechanism for approximate repeats functionally subsumes that of exact repeats. We then characterize this mechanism, revealing two main stages: PLMs first build feature representations using both general positional attention heads and biologically specialized components, such as neurons that encode amino-acid similarity. Then, induction heads attend to aligned tokens across repeated segments, promoting the correct answer. Our results reveal how PLMs solve this biological task by combining language-based pattern matching with specialized biological knowledge, thereby establishing a basis for studying more complex evolutionary processes in PLMs.
