Search papers, labs, and topics across Lattice.
This paper introduces MemLearner, a novel approach to enhancing Video World Models by integrating a learning-based adaptive context query method that utilizes query tokens to improve scene consistency over extended durations. The significance of this work lies in its ability to effectively handle scene occlusions and dynamic objects, which have been challenging for previous rule-based context retrieval methods. Experimental results show that MemLearner outperforms existing models, achieving superior memory utilization and scene consistency in complex video scenarios.
MemLearner achieves unprecedented scene consistency in video generation by learning to query context memory, outperforming traditional methods in dynamic and occluded environments.
Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame retrieval as memory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method using query tokens to bridge context and predicted tokens. By leveraging the video generation model itself for context querying, MemLearner exploits pre-trained visual priors without training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired with camera pose annotations, and propose a multi-dataset training strategy leveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms prior video world models in terms of scene consistency and memory, particularly under challenging occlusion and dynamic scenarios.
