Search papers, labs, and topics across Lattice.
This paper introduces Orthogonal JEPA, a novel framework for latent world modeling that employs orthogonal predictive factorization to enhance the learning of latent states in complex systems. By utilizing learned basis matrices and dedicated prediction branches for each component, the framework effectively reduces redundancy and improves gradient flow, leading to better representation quality and predictive performance. Experiments across diverse domains, including vision and health records, demonstrate significant improvements in forecasting and planning capabilities compared to standard joint-embedding predictive architectures.
Orthogonal JEPA achieves superior predictive performance by factorizing latent states, allowing for more efficient learning in complex systems.
World models construct latent states that support prediction, planning, and reasoning about an underlying system. Joint-embedding predictive architectures (JEPAs) offer a direct way to learn such states by predicting targets in representation space instead of reconstructing every detail of the observation. Standard JEPAs, however, organize all predictable content through one target embedding and one prediction pathway. In complex systems, this monolithic state can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure. We introduce \method, a latent world-modeling framework based on orthogonal predictive factorization. Learned basis matrices analyze each target state into multiple components, and a dedicated prediction branch estimates each component from a shared context representation. Predictive regression preserves the factor magnitudes required for state synthesis, an orthogonality objective discourages repeated directions, factor-activity regularization maintains variation in projected targets, and online variance regularization discourages coordinate-wise encoder collapse. Predicted components are synthesized into a complete latent state that can be used by a readout, decoder, planner, or autoregressive rollout. The same predictive-state mechanism applies when the target is temporally future, spatially hidden, or another partial observation of the same system. Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
