Search papers, labs, and topics across Lattice.
This paper introduces a sensorimotor world model (SMWM) that employs inverse dynamics regularization to train latent representations that are both stable and action-aligned. By addressing the common issues of representation collapse and the need for predictive accuracy, SMWM effectively biases the model towards controllable environmental factors while filtering out irrelevant information. The results demonstrate that SMWM achieves compact and interpretable latent spaces, leading to competitive planning performance in various control tasks without relying on complex training techniques.
Action-aligned representations can be achieved without complex training methods, enabling robust planning in dynamic environments.
Perception for action suggests that representations of the world should be shaped not by visual fidelity alone, but by their relevance for actions. At the same time, latent JEPA-style world models advocate learning compact predictive states from high-dimensional observations to facilitate the prediction of future states, but end-to-end training of these models is nontrivial because representations may collapse if our only goal is to construct a latent state that is easy to predict. We introduce a sensorimotor world model (SMWM): a latent world model trained end-to-end with inverse dynamics regularization. This single regularizer addresses both issues: it prevents representation collapse and induces action-aligned representations. By forcing latent states to preserve information about the action underlying a transition, it biases the model toward the controllable degrees of freedom of the environment while discarding uncontrollable distractors. This yields stable latent world models trained from offline, reward-free trajectories, without frozen encoders, exponential moving averages, or complex latent regularizers. Empirically, SMWM learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
ETH