Search papers, labs, and topics across Lattice.
DeltaWorld, a generative video world model, is introduced that operates on "delta" tokens, which encode the feature difference between consecutive frames in a vision foundation model's feature space into a single continuous token. This reduces the spatio-temporal video representation to a 1D temporal sequence, achieving significant compression (e.g., 1024x reduction for 512x512 frames). By enabling tractable multi-hypothesis training, DeltaWorld generates diverse and realistic future predictions with 35x fewer parameters and 2000x fewer FLOPs compared to existing generative world models.
Ditch the computational bloat: DeltaWorld slashes parameters by 35x and FLOPs by 2000x while generating more realistic video futures.
Anticipating diverse future states is a central challenge in video world modeling. Discriminative world models produce a deterministic prediction that implicitly averages over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaTok, a tokenizer that encodes the VFM feature difference between consecutive frames into a single continuous"delta"token, and DeltaWorld, a generative world model operating on these tokens to efficiently generate diverse plausible futures. Delta tokens reduce video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence, for example yielding a 1,024x token reduction with 512x512 frames. This compact representation enables tractable multi-hypothesis training, where many futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Code and weights: https://deltatok.github.io.