Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To resolve the failure of camera-controlled video models to render moving objects under novel viewpoints, the authors develop DynaTokens, a test-time adaptation framework that learns localized scene dynamics from a few trajectory examples while keeping the base model frozen. This addresses a fundamental bottleneck in world modeling, where base models struggle to decouple global perspective shifts from spatially localized physical motions. Across VBench2 and WorldScore benchmarks, the method achieves a superior Pareto frontier between camera-following fidelity and dynamic realism compared to standard LoRA and block fine-tuning.
Camera-controlled world models don't need weight updates to master scene physics: optimizing a handful of test-time tokens via cross-attention outperforms LoRA and block fine-tuning at decoupling local object dynamics from global camera trajectories.
Video generation must account for two sources of motion, one induced by the observer's camera path and the other caused by scene dynamics. An ideal camera-controlled video model should account for both motions: let users move the camera while evolving the scene dynamics. While current models handle camera-induced motion well in static settings, they struggle for dynamic scenes: objects are static, move incorrectly, or degrade in generation quality. We introduce DynaTokens, a lightweight set of learnable scene-specific tokens that teach dynamics to an existing camera-controlled world model. Our method is motivated by a simple asymmetry between the two sources of motion: whereas camera motion affects the generated view globally, object dynamics are spatially localized. Through cross-attention, DynaTokens trains the learnable tokens from a few example trajectories for a scene while keeping the base model frozen, and enables dynamics under new query camera paths. DynaTokens achieves a better simultaneous dynamics-camera tradeoff on VBench2 and WorldScore evaluations than LoRA, block finetuning, and specialized trainable-layer baselines. Analyses of token attention, ablations, and motion temporality suggest that matching the trainable interface to the structure of the learning target is important for effective adaptation. Project website: https://glab-caltech.github.io/dynatokens/