Search papers, labs, and topics across Lattice.
The paper introduces $\mu_0$, a scalable world model that utilizes 3D interaction traces to facilitate robot learning without the need for embodiment-specific action labels. By employing a novel TraceExtract system to automatically derive 3D supervision from diverse video sources, $\mu_0$ predicts smooth trajectories for salient interaction points, resulting in a compact and embodiment-agnostic motion interface. Experimental results demonstrate that $\mu_0$ significantly outperforms existing models in both 2D and 3D trace prediction, achieving competitive performance with action-supervised models despite being pretrained without action labels.
3D traces can serve as a powerful, scalable representation for robot learning, outperforming traditional action-based models without requiring specific action labels.
World models that capture how actions induce physical change enable scalable robot learning without reliance on embodiment-specific action labels. Pixel-space video models provide broad visual priors but expend model capacity on dense appearance reconstruction, while direct action models require embodiment-specific labels that hinder scalability. We present $\mu_0$, a scalable world model based on 3D traces. Rather than predicting dense pixels or directly modeling actions, $\mu_0$ forecasts smooth 3D trajectories for salient interaction points such as objects, tools, hands, and contact regions, yielding a compact, embodiment-agnostic motion interface. To enable training from diverse video sources, our TraceExtract system automatically extracts 3D supervision by selecting keypoints, constructing globally aligned traces, and associating motion segments with hierarchical language captions. This TraceExtract supervision pretrains $\mu_0$ by combining a pretrained vision-language backbone with a modular trace expert, which represents each query via B-spline control points and predicts future traces. Experiments show that $\mu_0$ outperforms baselines in both 2D and 3D trace prediction, including trace prediction models and tokenized VLM methods. Because $\mu_0$ is frozen and reusable, it can be paired with action experts for downstream robot embodiments. Despite action-free pretraining, the resulting trace-conditioned policies achieve performance competitive with VLA models pretrained with action supervision, such as $\pi_0$. These results establish 3D traces as a scalable and transferable representation for cross-embodiment manipulation.
