Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of ProtBERT and ESM2 embeddings for predicting adeno-associated virus (AAV) capsid viability, a common bioengineering task involving localized sequence variations. They find that amino acid-level embeddings initially outperform sequence-level embeddings in supervised tasks, while the reverse is true for unsupervised tasks. Crucially, fine-tuning embeddings with task-specific labels, particularly sequence-level representations, yields the best predictive performance, highlighting the limitations of pre-trained embeddings when dealing with sparse or localized mutations typical in bioengineering datasets.
Pre-trained protein embeddings fall short in predicting AAV capsid viability without task-specific fine-tuning, especially when mutations are sparse or localized, challenging their direct applicability in typical bioengineering scenarios.
Effective representations of protein sequences are widely recognized as a cornerstone of machine learning-based protein design. Yet, protein bioengineering poses unique challenges for sequence representation, as experimental datasets typically feature few mutations, which are either sparsely distributed across the entire sequence or densely concentrated within localized regions. This limits the ability of sequence-level representations to extract functionally meaningful signals. In addition, comprehensive comparative studies remain scarce, despite their crucial role in clarifying which representations best encode relevant information and ultimately support superior predictive performance. In this study, we systematically evaluate multiple ProtBERT and ESM2 embedding variants as sequence representations, using the adeno-associated virus capsid as a case study and prototypical example of bioengineering, where functional optimization is targeted through highly localized sequence variation within an otherwise large protein. Our results reveal that, prior to fine-tuning, amino acid-level embeddings outperform sequence-level representations in supervised predictive tasks, whereas the latter tend to be more effective in unsupervised settings. However, optimal performance is only achieved when embeddings are fine-tuned with task-specific labels, with sequence-level representations providing the best performance. Moreover, our findings indicate that the extent of sequence variation required to produce notable shifts in sequence representations exceeds what is typically explored in bioengineering studies, showing the need for fine-tuning in datasets characterized by sparse or highly localized mutations.
