Search papers, labs, and topics across Lattice.
This paper investigates the role of future-prediction in world action models, revealing that it can be effectively removed during inference without significant performance loss. The authors introduce Privileged Foresight Distillation (PFD), a method that transfers a corrective signal from a teacher model to a student model, enhancing action denoising while maintaining a current-only inference approach. Empirical results demonstrate that PFD leads to substantial improvements on manipulation benchmarks while ensuring minimal latency, reframing the understanding of future information as a distillable correction rather than a predictive target.
Future information in world action models isn't just a target to predict; it's a compressible correction that can be distilled for improved performance.
World action models jointly predict future video and action during training, raising an open question about what role the future-prediction branch actually plays. A recent finding shows that this branch can be removed at inference with little to no loss on common manipulation benchmarks, suggesting that future information may act merely as a regularizer on the shared visual backbone. We propose instead that joint training induces an action-conditioned correction that privileged future observations impose on action denoising, and that current-only policies capture this correction only partially. Making the account precise, we formulate privileged foresight as a residual in the action-denoising direction -- the difference between what a model predicts given the true future and what it predicts given only the current frame -- and introduce \emph{Privileged Foresight Distillation (PFD)}, which transfers this residual from a training-time teacher into a small adapter on a current-only student. The teacher and student share the same backbone and differ only in the attention mask over video tokens; future video is never generated at inference. Controlled experiments verify that this gain reflects a genuine future-conditioned correction rather than a side effect of capacity or regularization. Empirically, PFD achieves consistent improvements on LIBERO and RoboTwin manipulation benchmarks while preserving the current-only inference interface at negligible added latency. This view reframes the role of future information in world action models: not as a target to predict, nor as a regularizer to absorb, but as a compressible correction to be distilled.
