Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the fidelity of action-conditioned world models in generating future states based on arbitrary actions, revealing that while these models perform well with expert actions, they falter with off-expert trajectories. The authors introduce WorldEcho to assess action following across a wider action distribution and find that current models either ignore commands or produce visually inconsistent outputs. To enhance performance, they propose WorldSync, which improves action following through broader training distributions, grounding video representations in robot dynamics, and aligning predicted outcomes with real-world changes, resulting in higher success rates in policy learning tasks.
Current action-conditioned world models fail to reliably follow diverse off-expert actions, risking the effectiveness of policy learning in real-world applications.
Action-conditioned world models are increasingly used as learned simulators for policy evaluation and improvement, yet their effectiveness rests on an unverified assumption: generated futures faithfully reflect arbitrary valid actions. Existing benchmarks are typically confined to expert demonstrations, leaving off-expert action following inadequately evaluated. To address this gap, we introduce WorldEcho, which probes action following over a broader action distribution using visual integrity and SE(3) trajectory alignment. Our diagnosis shows that current world models reasonably execute expert actions but struggle with diverse off-expert trajectories, either ignoring the commanded actions or producing visually invalid rollouts. We further propose WorldSync, which strengthens action following along three complementary axes: distributional coverage, representational grounding, and intervention-effect alignment. It broadens the training distribution over action consequences, grounds intermediate video representations in action-induced robot dynamics through an Action-Forcing Expert, and aligns predicted changes under action interventions with the corresponding changes in ground-truth futures. Experiments on RoboTwin benchmarks and real-robot tasks show that WorldSync improves WorldEcho metrics and serves as a more reliable simulator for iterative policy improvement, enabling policies to achieve higher success rates.