Search papers, labs, and topics across Lattice.
DriftWorld introduces a novel action-conditioned world model that leverages drifting generative models to generate future frames from current observations and action sequences in a single forward pass, achieving over 30 frames per second. This approach significantly reduces the inference time, making it 17 times faster than traditional diffusion-based models, which struggle with multistep sampling. Evaluated on various robotic manipulation benchmarks, DriftWorld not only excels in decision-making performance but also demonstrates high correlation with ground truth in offline policy ranking, highlighting its effectiveness in both real-time planning and evaluation tasks.
DriftWorld achieves 17x faster rollouts than diffusion models, revolutionizing real-time planning for robotic manipulation.
Predictive world models enable robots to plan by imagining the outcomes of their actions, but their value for control hinges on generating many rollouts quickly. This creates a bottleneck for diffusion-based world models: multistep sampling makes each rollout expensive, limiting large-scale action search at inference time. We introduce DriftWorld, an action-conditioned world model based on drifting generative models. Rather than denoising iteratively at inference, DriftWorld learns an action-conditioned drift during training, allowing it to generate future frames from the current observation and a candidate action sequence in a single forward pass at 30+ fps, which is 17x faster on average than diffusion based baselines. We evaluate DriftWorld on standard vision-based robotic manipulation benchmarks, including Bridge-V2, RT-1, Language Table, Push-T, and Robomimic. By producing rollouts that are both accurate and fast, DriftWorld achieves state-of-the-art decision-making performance with far less inference time than diffusion-based world model baselines. Beyond online control, DriftWorld can also serve as an offline simulator for ranking real-world robot policies, with rollout-based scores correlating with ground truth at up to 0.99. These results show that drifting models are a strong fit for robot world modeling, where fast, high-quality imagination directly supports planning and policy evaluation.
MIT CSAIL