Search papers, labs, and topics across Lattice.
This paper introduces GaussianWAM, a novel framework that enhances World-Action Models (WAMs) by integrating geometric and semantic supervision through 3D Gaussian fields during training. By leveraging synchronized multi-view observations and frozen models for depth and semantic features, GaussianWAM effectively organizes these heterogeneous signals into shared Gaussian primitives, significantly improving the model's ability to predict future visual states and generate actions. The results demonstrate substantial performance gains on the LIBERO-Plus benchmark, with FastWAM and Cosmos Policy achieving improvements of 19.24% and 5.78%, respectively, showcasing the effectiveness of this representation-enhancement approach.
Training-time Gaussian distillation can boost WAM performance by over 19%, transforming how we integrate geometry and semantics in robotic manipulation.
World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose \textbf{GaussianWAM}, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05\% to 71.29\% and Cosmos Policy from 71.52\% to 77.30\%. Direct CLIP and VGGT distillation already establishes a strong FastWAM baseline of 69.37\%, while Gaussian-field unification further improves it to 71.29\%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.
