Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper introduces recAMPLIFY, a recursive pretraining approach for masked protein language models that refines residue-aligned hidden states over multiple cycles using a weight-tied Transformer core. By adapting recursive pretraining to masked encoders, recAMPLIFY achieves competitive performance with significantly fewer parameters than previous models, demonstrating improved contact recovery and fold accuracy. The authors further propose Gated Mixture Recurrence (GMR) to adaptively control computation depth during decoding, unifying halting and gating mechanisms and reducing expected layer depth without sacrificing accuracy.
Recursive pretraining with gated mixture recurrence unlocks substantial parameter efficiency and adaptive computation for protein language models, outperforming larger models on key structural prediction tasks.
Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.