Search papers, labs, and topics across Lattice.
ShadowDancer introduces a novel framework for achieving any-action, frame-level control in interactive video world models by leveraging unified dynamics representations. The method overcomes the limitations of existing action encoding systems by utilizing shadow pairs鈥攙ideo pairs that capture the same dynamics under varied appearances鈥攁nd cross-shadow prediction to enhance action transfer across different scenes. Experimental results indicate a significant improvement in action transfer and long rollout performance, achieving an average blinded win rate of 86% against strong baselines.
ShadowDancer enables seamless action transfer across diverse environments by transforming any demonstrated video clip into a reusable action asset without the need for action labels or fine-tuning.
We present ShadowDancer, a novel approach to any-action, frame-level control of interactive video world models. The obstacle is representational: existing interfaces either encode an action loosely, leaving how it unfolds for the model to improvise, or encode it exactly through structured signals that serve one family and are hard to acquire, so precise control across diverse dynamics remains impractical. Demonstration videos are the natural remedy, specifying any dynamics frame by frame; yet a video shows its dynamics only through one particular appearance, a single shadow of the underlying dynamics, so actions learned from demonstrations transfer poorly to new scenes. ShadowDancer addresses this with two key innovations: (1) shadow pairs, video pairs that replay the same dynamics under independently resampled appearance, constructed at scale by our Shadow Library, so that a dynamics family becomes controllable exactly when such pairs can be constructed for it; and (2) cross-shadow prediction, which learns actions by predicting one shadow from the other, so that whatever the pairing resamples is discarded by construction and whatever it preserves becomes the action, yielding a unified dynamics representation that drives a block-causal world model. Any demonstrated clip thus becomes a reusable action asset, replayed in new environments without action labels, motion estimators, or fine-tuning. Experiments demonstrate improved action transfer and long action rollout over strong latent-action and interactive world model baselines across diverse dynamics families, with an average blinded win rate of 86% in rollout comparisons. We show video results at https://ShadowDancer-1.github.io
