Search papers, labs, and topics across Lattice.
This study systematically evaluates the performance of transformer-based models, specifically DNABERT2, against traditional convolutional architectures like ConvNova in the context of genomic sequence decoding. By addressing the effectiveness of extensive pretraining and the implications of Byte Pair Encoding (BPE) tokenization, the authors reveal that while transformer models show performance improvements, the benefits may not always justify the high pretraining costs. The findings indicate that BPE tokenization's relevance in genomic tasks requires further scrutiny, as its impact on model performance is not as clear-cut as previously assumed.
Transformer models in genomics may not always deliver the expected performance gains relative to their pretraining costs, challenging the status quo in DNA sequence analysis.
Recent breakthroughs in foundation models and Large Language Models (LLMs) have introduced new opportunities for studying and decoding genomic sequences. Several state-of-the-art approaches, such as DNABERT2, rely on transformer-based architectures, while others, such as ConvNova, still build upon more conventional convolutional models. However, systematic benchmark comparisons across these methods remain scarce. Given that transformer-based models require extensive and costly pretraining, it is crucial to evaluate whether their performance gains justify this overhead. Moreover, LLMs such as DNABERT2 typically rely on Byte Pair Encoding (BPE) tokenization, whose relevance for DNA sequence representation is still debated within the genomics community. In this work, we investigate three key questions: (i) do transformer-based models provide sufficient improvements on fine-tuning tasks upon heavy pretraining, (ii) what is the actual contribution of pretraining in this setting, and (iii) how does BPE tokenization impact performance on genomics-related tasks?
