Search papers, labs, and topics across Lattice.
This paper critiques the prevailing notion that video generation inherently equates to world modeling, arguing instead that existing models only capture a partial and implicit understanding of spatiotemporal dynamics. The authors introduce the concept of counterfactual controllability, which emphasizes the ability of models to reason about the consequences of actions and to adapt their generative processes based on this reasoning. The key finding is that incorporating counterfactual reasoning into video generation can lead to more robust and self-evolving world models, enhancing their applicability in real-world scenarios.
Counterfactual controllability in video generation could be the key to creating self-evolving world models that understand and adapt to their actions.
Existing literature claims that video generation essentially is world modelling. On the one hand, the claim is productive because it pushes generative AI beyond static images and toward temporally extended physical scenes. On the other hand, this claim dangerously relies on the belief that scaling visual prediction alone will automatically yield physical agents. We prefer a more accurate statement: video generation models learn a partial, implicit spatiotemporal world model, but not a fully grounded or controllable one. The reason is as follows: a model may generate a plausible video of a drone crossing a forest or a robot arm manipulating a cup, yet still fail to know which variables are controllable, which constraints belong to a particular body and which futures remain valid under intervention. The frontier in essence is not predictive realism alone, instead it emphasizes a self-evolving generative nature that requires the decisive criterion to be counterfactual controllability: the capability of asking what would happen under an action, to test whether the generated future can survive embodiment constraints and to feed the resulting action knowledge back into future imagination (generation). Therefore, in this paper we present a new perspective, i.e., autonomous video generation with counterfactual controllability is one promising way to realize self-evolving world models.
Tsinghua AI