Search papers, labs, and topics across Lattice.
This paper introduces DepthWorld, a 3D world model for robot manipulation that addresses the inconsistency of current video-based models by incorporating explicit 3D geometric supervision. By developing a calibration pipeline to generate DROID-3D, a dataset with dense metric depth and recalibrated multi-view extrinsics, they enable training a Stable Video Diffusion-based model that jointly predicts RGB and depth. This approach not only improves RGB prediction accuracy but also provides the necessary geometric fidelity for downstream manipulation tasks.
Robot world models can now predict photorealistic RGB *and* accurate metric depth, bridging the gap between visually plausible rollouts and geometrically consistent 3D environments.
World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving<0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.
