Search papers, labs, and topics across Lattice.
This paper introduces In-Context World Modeling (ICWM), a novel framework that enables robotic control systems to autonomously identify and adapt to their operational context by leveraging a history of self-generated interactions. By treating system identification as an in-context adaptation problem, ICWM allows robots to infer essential system variables without the need for extensive fine-tuning or parameter updates, thus enhancing generalization to novel environments. Experimental results show that ICWM significantly outperforms traditional Vision-Language-Action models, particularly in scenarios with altered camera viewpoints and robot configurations.
Robots can now autonomously adapt to new environments without retraining, using just a history of their own interactions.
Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.
