Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
LeWAM, a bidirectional transformer trained end-to-end on a decoder-free JEPA latent, addresses the noise and redundancy inherent in reconstruction-based world action models by jointly predicting forward, backward, and inverse dynamics alongside policy. This approach achieves state-of-the-art alignment by enabling linear probes to extract robot and object states from the latent while ignoring visual distractors, and demonstrates competitive closed-loop performance with flow-matching policies. Furthermore, LeWAM's planning mechanism, which samples actions in the policy head's noise space, mitigates inaccuracies in the dynamics model for improved MPC performance.
A single JEPA latent space unifies state representation, inverse dynamics, and policy prediction, outperforming prior reconstruction-based world action models in state alignment and achieving competitive closed-loop control.
World action models (WAMs) predict actions and future observations, typically from a reconstruction-based representation that carries noisy, redundant information which can complicate downstream predictions. We introduce LeWAM, a bidirectional transformer for forward, backward, inverse dynamics and policy prediction, on a decoder-free JEPA latent trained end-to-end through all four modes. We see the following benefits: 1) Alignment: linear probes read robot and object state from LeWAM's latent better than from a regular Le World Model (a forward-only JEPA world model), while the latent ignores visual distractors as well as LeWM does and far better than a reconstruction-based WAM. 2) Acting: Closed-loop evaluations of LeWAM match a regular flow-matching policy trained on the same encoder at matched size, while also providing a world model. 3) Planning: Sampling raw actions when planning with WAMs lets MPC exploit dynamics-model inaccuracies; planning in the noise space of the policy head instead improves the closed-loop performance of these WAMs.
NVIDIA