Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce ZEST, an evolution-informed tokenization strategy that embeds biological priors by leveraging conserved regions from multiple sequence alignments, addressing the limitations of standard tokenization for protein sequences. This approach enables a compressed representation, allowing a 200M-parameter model (LEMON) to process significantly longer sequences within typical context windows. LEMON achieves state-of-the-art performance in remote homology detection, outperforming larger models and demonstrating that effective tokenization can rival massive parameter scaling for efficient, biologically-grounded protein representation learning.
Evolution-informed tokenization allows a 200M-parameter protein model to outperform 3B-parameter giants by embedding biological priors directly into the vocabulary.
Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.