Search papers, labs, and topics across Lattice.
This paper introduces LAWM-3D, a novel approach that enhances latent action models (LAMs) by integrating multi-view video inputs to learn 3D-aware latent actions for robot world models. Despite the initial hypothesis that multi-view data would improve action representation, the study reveals that challenges such as future-frame appearance leakage and inter-camera discrepancies hinder performance. The proposed method incorporates a unified action tokenization scheme, geometric alignment constraints, and a non-injective RGB-D reconstruction objective, leading to significant advancements in generation quality, physical consistency, and generalization in world models.
Simply adding multi-view videos to latent action models doesn't guarantee 3D awareness; LAWM-3D reveals the critical design choices needed for success.
World models enable agents to perform forward rollout and planning without real-world interaction. However, their application in open-world embodied intelligence remains limited by the high cost of action annotations and the heterogeneity of action spaces across platforms. Recently, latent action models (LAMs) have alleviated this bottleneck by learning action representations directly from unlabeled human videos in a self-supervised manner. Nevertheless, most existing LAMs rely on single-view inputs and operate primarily in 2D pixel space, raising a fundamental question: can simply incorporating multi-view videos into LAM training endow the learned latent actions with 3D-aware perception? Our study shows that the answer is negative. The primary reasons lie in future-frame appearance leakage as well as inter-camera appearance discrepancies and viewpoint variations. To address these issues, we propose LAWM-3D, which introduces three tightly coupled key designs: (1) a multi-view invariant unified action tokenization scheme for learning 3D-aware latent actions; (2) a geometric alignment constraint that anchors intermediate encoder features to a pretrained 3D foundation model, thereby explicitly providing cross-view geometric correspondences; and (3) a non-injective RGB-D joint reconstruction objective that prevents shortcut learning from future-frame appearance information, forcing the LAM to focus supervision on motion cues with geometric significance. Importantly, these components are not simply stacked but are tightly coupled through a unified motivation. Built upon a two-stage paradigm of large-scale human video pretraining followed by robot fine-tuning, extensive experiments demonstrate that the proposed 3D-aware latent actions significantly improve world model performance, achieving SOTA results in generation quality, physical consistency, and generalization ability.
