Search papers, labs, and topics across Lattice.
This paper introduces the Semantic-Temporal World Action Model (ST-WAM), which enhances the robustness of robot manipulation under visual distribution shifts by leveraging DINOv3 features for future prediction and history retrieval. The authors identify and address the issue of Training-Distribution Hallucination, where models incorrectly predict future states based on training-domain content rather than the current visual context. ST-WAM achieves significant performance improvements, including a 21.3 percentage point increase in zero-shot LIBERO-Plus performance and a doubling of real-world success rates under visual shifts, demonstrating the effectiveness of semantic-temporal modeling in action prediction.
Training-Distribution Hallucination is a critical challenge in robot manipulation, but ST-WAM's innovative use of DINOv3 features dramatically boosts performance under visual shifts.
World Action Models (WAMs) have emerged as a promising paradigm by jointly modeling robot actions and future visual dynamics. However, their reliance on pixel-generative future supervision can entangle action-relevant state transitions with task-irrelevant visual content, limiting robustness under visual distribution shifts. We identify Training-Distribution Hallucination, a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene. A controlled frame-triplet diagnosis further shows that DINOv3 features remain more stable across visual shifts while better preserving task-state distinctions than Wan-VAE latents. Rather than correcting the predicted futures, we propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics. Its Dual-Space Future Experts (DSFE) jointly predict future VAE latents and DINO features, while Current-Anchored Intent Retrieval (CAIR) retrieves task-relevant evidence from recent DINO history under the current visual-language context. ST-WAM is trained end-to-end without additional embodied pretraining or task-specific annotations, and requires no explicit future generation at inference. It achieves 98.7% on LIBERO and 92.8% on RoboTwin 2.0; more importantly, compared with Fast-WAM, it improves zero-shot LIBERO-Plus performance by 21.3 percentage points and more than doubles real-world success under visual shifts from 25.8% to 61.5%. These results demonstrate that semantic-temporal modeling effectively complements pixel-generative dynamics for robust manipulation.
