Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
Proxy2World is a controllable world model that learns to synthesize dynamic scenes from lightweight, coarse proxies without requiring paired proxy-video supervision. By training on ordinary posed RGBD videos via cross-modal flow matching, the framework jointly learns depth-conditioned RGB synthesis and joint RGBD generation to bypass data curation bottlenecks. Benchmarked on ProxyBench, its hybrid proxy-camera denoising strategy outperforms conventional camera- and geometry-conditioned baselines by maintaining faithful structural alignment without sacrificing visual fidelity.
Video world models can now faithfully respect coarse, user-designed 3D scene proxies without ever training on a single hand-crafted proxy-video pair.
Lightweight scene proxies let creators control scene layout and motion while leaving room for imagination in appearance, lighting, and visual effects. However, a suitable proxy is not uniquely defined, making paired proxy-video data difficult to construct automatically at scale. We present Proxy2World, a controllable world model that learns these complementary capabilities from ordinary posed RGBD videos, without training on authored proxy-video pairs. The model jointly learns depth-conditioned RGB generation and joint RGBD generation through cross-modal flow matching. Learning both tasks enables proxy-camera hybrid denoising at inference to follow the proxy structure while producing natural, detailed visuals. We further introduce ProxyBench to evaluate this capability across a diverse set of scenes, camera trajectories, and subject motions. Experiments on ProxyBench show that Proxy2World achieves a better balance between structural adherence and visual quality than camera-controlled and geometry-conditioned methods, supported by quantitative metrics, VLM assessments, human evaluations and diverse qualitative results.