Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the loss landscape of transformer models trained for protein structure prediction using Langevin dynamics to sample at varying temperatures. They find that transformers, unlike feedforward networks, exhibit a range of intermediate temperatures with good learning properties due to the absence of a first-order-like transition in the loss. The study also identifies an optimal embedding dimension where layer parameters are highly conserved and demonstrates that attention matrices at higher temperatures and embedding dimensions are more predictive of protein contact maps.
Forget hyperparameter sweeps: the secret to better protein structure prediction with transformers may lie in sampling the loss landscape at the right "temperature."
We investigate the parameter space of transformer models trained on protein sequence data using a statistical mechanics framework, sampling the loss landscape at varying temperatures by Langevin dynamics to characterize the low-loss manifold and understand the mechanisms underlying the superior performance of transformers in protein structure prediction. We find that, at variance with feedforward networks, the lack of a first--order--like transition in the loss of the transformer produces a range of intermediate temperatures with good learning properties. We show that the parameters of most layers are highly conserved at these temperatures if the dimension of the embedding is optimal, and we provide an operative way to find this dimension. Finally, we show that the attention matrix is more predictive of the contact maps of the protein at higher temperatures and for higher dimensions of the embedding than those optimal for learning.