Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of 4-bit quantization and QLoRA on various protein language models (PLMs) for both prediction and generation tasks. The study finds that these efficiency techniques can significantly reduce GPU memory requirements (up to 90%) while often retaining over 90% of full fine-tuning performance on prediction tasks, though results vary by model and configuration. For generative models, while structural and sequence-level properties are largely preserved, distributional analysis reveals subtle, model-dependent shifts in autoregressive outputs.
Protein language models can be adapted with QLoRA and 4-bit quantization, slashing memory needs by up to 90% while retaining >90% prediction performance on many tasks.
Background: Protein language models (PLMs) are increasingly used for sequence generation and property prediction, but their size makes fine-tuning and deployment expensive. The effects of quantization and parameter efficient fine-tuning on performance, representations and generation remain insufficiently characterized. Results: We evaluated 4-bit quantization and low-rank adapter fine-tuning (QLoRA) across ESM-2, ESMC, ProtBERT, ProtT5, Ankh, Ankh3 and Profluent-E1. Across protein prediction tasks, many model-task pairs retained more than 90% of full fine-tuning performance. Peak GPU memory savings approached 90% for the largest models, although performance and efficiency varied by model, dataset and training configuration. QLoRA often preserved early-layer representations while inducing task-specific adaptations in middle and late layers, resembling full fine-tuning with smaller representational changes. Training speed and power effects were more varied. For unconditional generation with ProLLaMA, ProtGPT2, ProGen2, ProteinGLM and ESM3, 4-bit quantization largely preserved predicted structural and sequence-level properties, but token-level analysis revealed model-dependent shifts in autoregressive output distributions. Conclusion: QLoRA and 4-bit quantization reduce PLM computational requirements, particularly GPU memory usage. Our results support QLoRA as a first-pass strategy for memory limited adaptation, reserving full fine-tuning for challenging tasks, unstable architectures or low validation recovery. For generative PLMs, sequence-level and structural metrics should be complemented with distributional analysis, since downstream predictions alone may miss quantization-induced shifts. These approaches can broaden access to large-scale protein modelling while requiring model- and task-specific validation.