Search papers, labs, and topics across Lattice.
This paper introduces PhyLatent, a novel training objective designed to enhance the performance of Joint-Embedding Predictive Architecture (JEPA) world models by addressing three critical failure modes: physical invariance collapse, physical identifiability collapse, and counterfactual dynamics collapse. By implementing targeted training pathways, the authors significantly reduce failure rates on the OGBench-Cube benchmark and improve model predictive control success rates across various environments. The results indicate that merely preventing global latent collapse is inadequate for developing robust world models, underscoring the importance of dynamics-relevant representations.
PhyLatent slashes failure rates in JEPA world models while boosting predictive control success from 70% to 78%鈥攁nd even up to 98% in complex environments.
We propose PhyLatent, a dynamics-relevant training objective for JointEmbedding Predictive Architecture (JEPA) world models. Our key observation is that preventing global latent collapse does not ensure that a representation preserves physical states and action consequences. We identify three failure modes in JEPA world models: physical invariance collapse, physical identifiability collapse, and counterfactual dynamics collapse. PhyLatent addresses them through three training pathways: physical invariance, physical identifiability, and counterfactual dynamics, implemented with physical state grounding, future representation alignment, static visual invariance, counterfactual branch separation, and latent denoising. On OGBench-Cube, PhyLatent reduces the three failure rates from 15.60%, 6.71%, and 8.41% to 7.53%, 0.95%, and 4.62%, respectively, and improves model predictive control (MPC) success from 70.0% to 78.1%. With the same architecture and planner, it further improves success from 81.0% to 98.0% on TwoRooms and remains competitive on Reacher and PushT. These results show that global non-collapse alone is insufficient for learning a reliable JEPA worldmodel state space.
