Search papers, labs, and topics across Lattice.
The paper introduces JEPA-DNA, a pre-training framework for Genomic Foundation Models (GFMs) that combines Joint-Embedding Predictive Architecture (JEPA) with generative objectives to capture both local syntax and global functional context. JEPA-DNA uses a CLS token to predict high-level functional embeddings of masked genomic segments in the latent space, thereby grounding the model in biological function. Evaluations show JEPA-DNA consistently outperforms generative-only baselines in supervised and zero-shot genomic tasks, leading to more robust and biologically grounded representations.
Genomic foundation models can now capture broader functional context and achieve superior performance in genomic tasks by integrating a Joint-Embedding Predictive Architecture (JEPA) that predicts high-level functional embeddings.
Genomic Foundation Models (GFMs) have largely relied on Masked Language Modeling (MLM) or Next Token Prediction (NTP) to learn the language of life. While these paradigms excel at capturing local genomic syntax and fine-grained motif patterns, they often fail to capture the broader functional context, resulting in representations that lack a global biological perspective. We introduce JEPA-DNA, a novel pre-training framework that integrates the Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. JEPA-DNA introduces latent grounding by coupling token-level recovery with a predictive objective in the latent space by supervising a CLS token. This forces the model to predict the high-level functional embeddings of masked genomic segments rather than focusing solely on individual nucleotides. JEPA-DNA extends both NTP and MLM paradigms and can be deployed either as a standalone from-scratch objective or as a continual pre-training enhancement for existing GFMs. Our evaluations across a diverse suite of genomic benchmarks demonstrate that JEPA-DNA consistently yields superior performance in supervised and zero-shot tasks compared to generative-only baselines. By providing a more robust and biologically grounded representation, JEPA-DNA offers a scalable path toward foundation models that understand not only the genomic alphabet, but also the underlying functional logic of the sequence.
