Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of off-manifold collapse in guided protein language models, where overly strong guidance leads to degenerate protein sequences that are statistically indistinguishable from random inputs. The authors identify a detectable signature of this collapse, allowing for the differentiation between successful and failed generations based on structural and compositional properties. By implementing a post-hoc filtering method called Mahalanobis filtering, they enhance the quality of generated sequences while maintaining the original generator's integrity, resulting in improved property scores and structural plausibility.
Guided protein language models can produce low-complexity sequences that fool property oracles, but a simple filtering technique can recover meaningful outputs without retraining.
Protein language models are widely used priors for protein sequence design, and a growing body of work controls them at inference time as an alternative to fine-tuning. Such guidance faces a dilemma: mild enough to preserve natural activation statistics, it barely moves the property; strong enough to move it, the generations become progressively harder to fold. We show the failure has a specific and cheaply detectable signature, an off-manifold collapse of the model's own representations. Guided activations fall toward a region statistically indistinguishable from random amino-acid input, and the sequences degenerate to low complexity, yet the property oracle being optimized can still score these generations as a success. The optimized oracle can therefore fail to witness the collapse and, for solubility, can actively reward it, whereas structure and composition expose the failure. Because the failure is already visible in a finished candidate, we detect it at the output rather than modify the generator. We introduce a cheap density prior over natural protein activations and keep only the candidates that remain typical under it, a training-free post-hoc step we call Mahalanobis filtering. At matched guidance settings it improves both the property score and the structural plausibility of the sequences it keeps at negligible cost, without touching the generator, and transfers across different guidance methods. We release the activation statistic at https://huggingface.co/Shuibai12138/off-manifold-collapse-plm
