Search papers, labs, and topics across Lattice.
This paper introduces Flex-$\pi$, a multi-stream world-action model that leverages a frozen video-generation VAE to encode both RGB data and 3D pointmaps without additional training. By integrating 3D geometry and object semantics into its training regime, Flex-$\pi$ achieves significant improvements in demonstration efficiency and generalization on complex bimanual manipulation tasks. The model outperforms existing baselines by 2-7脳 while maintaining faster inference speeds, showcasing a novel approach to multi-modal learning in robotics.
Flex-$\pi$ achieves unprecedented efficiency in bimanual manipulation by jointly learning from RGB and 3D geometry without extra training costs.
World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-$\pi$, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7$\times$ on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than $\pi_{0.5}$. Our project website: https://flex-pi.github.io/
AI2
UW