Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces function-aware masking for antibody-specific language models, enhancing the pretraining process by aligning mask placement with functional priors derived from IMGT annotations and structural predictions. By leveraging these tailored masking strategies, the authors achieve significant performance improvements, including up to a 14% gain on structure-related tasks and a 5.9x enhancement on CDR-related tasks. The findings highlight the effectiveness of informed mask placement as a parameter-free method to impose functional inductive biases in the training of antibody language models.
Function-aware masking can boost antibody model performance by up to 14% by strategically aligning mask placements with biological functions.
Antibody-specific language models pretrained via masked language modeling (MLM) learn representations that are critical for downstream sequence design and property prediction tasks. Yet, the corruption process itself is rarely leveraged as a source of inductive bias during pretraining. While preferentially masking complementarity-determining regions (CDRs) improves binding-related predictions, antibodies possess diverse biological priors over a variety of functions. Herein, we introduce function-aware masking, a family of pretraining algorithms that align mask placement with specific functional priors (e.g., from IMGT annotations or structure predictions) to shape the learned representation space. We show that these specialist masking strategies significantly improve performance on their respective objectives, yielding up to a 14% gain on structure-related tasks and up to a 5.9x improvement on CDR-related tasks. To further improve performance across multiple functional axes, we develop hybrid masking strategies that integrate multiple priors, balancing reconstruction over binding, structural, and biophysical objectives. Our results demonstrate that informed mask placement provides a parameter-free mechanism for imposing functional inductive biases in antibody language model training.