Search papers, labs, and topics across Lattice.
This paper establishes a generalization theory for Joint Embedding Predictive Architectures (JEPAs) by framing JEPA pretraining as a conditional spectral graph learning problem. The authors demonstrate that the JEPA objective corresponds to a low-rank factorization of an action-conditioned co-occurrence matrix, linking pretraining error to downstream planning regret. Their findings reveal a trade-off between approximation and sample errors in relation to latent dimensions, enhancing the theoretical understanding of JEPA-based world models compared to traditional input-level predictive methods.
JEPA-based world models face a fundamental trade-off between approximation and sample errors that could redefine their application in predictive tasks.
Joint Embedding Predictive Architectures (JEPAs) have recently emerged as a promising paradigm for world modeling by learning predictive dynamics in a latent space rather than generating future observations at the input level. Despite their empirical success, the theoretical understanding of JEPA-based world models remains limited. In this paper, we develop the first generalization theory for JEPA-based world models. We formulate JEPA pretraining as a conditional spectral graph learning problem and show that the JEPA objective is equivalent to a low-rank factorization of an action-conditioned co-occurrence matrix. Building on this characterization, we establish a connection between JEPA pretraining error and downstream planning regret, leading to a finite-sample generalization bound for JEPA-based world models. Our analysis reveals an inherent trade-off between approximation and sample errors with respect to the latent dimension, providing theoretical insights into the advantages and limitations of latent predictive models compared with input-level predictive approaches.
