Search papers, labs, and topics across Lattice.
This paper introduces WALA, a framework that learns executable latent actions by leveraging both action-labeled demonstrations and action-free videos, addressing the challenge of collecting scalable robot action data. By pretraining a semantic-geometric latent action model that predicts future deltas in a feature space, WALA effectively captures task-relevant structures while minimizing sensitivity to visual details. Experimental results demonstrate that WALA not only sets a new state-of-the-art on RoboCasa with a 75.2% average success rate but also enhances policy performance and generalization in real-world manipulation tasks.
WALA achieves a remarkable 75.2% average success rate on RoboCasa, showcasing the power of combining action-labeled and action-free data for robust robot learning.
Generalizable robot policies typically rely on action-labeled robot demonstrations, which are expensive to collect and difficult to scale. In contrast, large-scale human and robot videos contain rich physical interactions but often lack executable robot action labels. We present WALA, a framework for learning executable latent actions from both action-labeled demonstrations and action-free videos. WALA first pretrains a semantic-geometric latent action model from videos by modeling the evolution between current observations and sparsely sampled future observations. Instead of reconstructing raw pixels, WALA predicts future deltas in the DINOv3 feature space and dense depth space, preserving task-relevant semantic and geometric structure while reducing sensitivity to appearance details. During policy training, the pretrained encoder provides stable latent action targets, and the decoder serves as a trainable latent world model. The latent actions generated by the vision-language backbone are jointly supervised by robot action prediction, latent action target matching, and future dynamics prediction. This enables action-labeled demonstrations to provide executable control supervision, while action-free videos contribute dynamics supervision without requiring robot action annotations. Experiments show that WALA achieves strong performance on RoboTwin, sets a new state-of-the-art result on RoboCasa with 75.2% average success, and improves both policy performance and generalization in real-world manipulation tasks.
