Search papers, labs, and topics across Lattice.
This paper investigates the video-action generalization gap observed in video-action models (VAMs) and world-action models (WAMs) after fine-tuning on robotic action data, where compositional priors from generative video foundation models are often lost. The authors introduce the Temporal Ratio (TR), an attention-based metric that quantifies a model's reliance on future latent rollouts, revealing its predictive power for compositional generalization capacity. By leveraging TR, they propose an adaptive guidance method that enhances compositional conditioning signals during inference, effectively addressing the out-of-distribution and in-distribution compositional generalization gap, as demonstrated on the LIBERO benchmark and real-world tasks.
The Temporal Ratio reveals how attention shifts between future and present frames can predict a model's ability to generalize compositional tasks in video-action contexts.
Generative video foundation models exhibit strong compositional priors, yet world-action models (WAMs) and video-action models (VAMs) often lose these priors after finetuning on robotic action data. We refer to this discrepancy as the video-action generalization gap. In this paper, we systematically investigate this gap by evaluating a comprehensive design space of VAMs, demonstrating that standard design choices yield no emergent explanation pattern. To explain this behavior, we introduce the Temporal Ratio (TR), an attention-based measure of how strongly the action head relies on future latent rollouts relative to the anchored current frame. TR has two key properties: first, a model's structural reliance on future-predictive latents, measured via TR, acts as a predictor of its compositional generalization capacity; second, it natively fluctuates based on task phase, shifting attention to future frames during planning and reverting to the present frame for precise manipulation. Finally, based on these findings, we propose an inference-time adaptive guidance method, which exploits this intrinsic feature attention pattern to dynamically amplify compositional video conditioning signals precisely when the policy relies on future rollouts. Evaluated on the LIBERO benchmark and real-world tasks, our approach mitigates the OOD-ID compositional generalization gap. More details: https://umishra.me/temporal-ratio/
Tsinghua AI