Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
This paper introduces PPMem, a video world model that leverages persistent planar primitives as its 3D memory to generate and explore environments at scale. By augmenting a video diffusion transformer with a planar prediction head and a Depth Token Fusion module, PPMem jointly predicts RGB video and planar geometry, merging them into a compact, geometrically explicit, and cross-view consistent memory. The model achieves state-of-the-art performance on DL3DV and RealEstate10K, outperforming prior 3D-grounded models in video quality while generating metric-scale depth, surface normals, and exportable planar geometry that remains coherent over hundreds of frames and large viewpoint changes.
A single scene can be represented by ~100K plane primitives (5MB), enabling video world models to maintain geometric consistency across hundreds of frames and large viewpoint changes.
Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced. Plane primitives offer a natural candidate: a compact set of oriented planes captures the dominant structural elements (e.g., walls, floors, facades) of indoor and outdoor scenes, while requiring only 7 parameters per primitive. We present PPMem, a video world model that adopts plane primitives as its persistent 3D memory. PPMem augments a video diffusion transformer with a planar prediction head and a Depth Token Fusion (DTF) module: each autoregressive chunk jointly produces RGB video and planar geometry, which merges into a persistent plane memory that conditions subsequent chunks via rendered depth. The resulting memory is three orders of magnitude more compact than per-pixel point clouds (~100K primitives, 5MB for a full scene), geometrically explicit (metric depth, normals, exportable surfaces), cross-view consistent (jointly predicted by the shared backbone), and adaptive (refined by the merging procedure as generation proceeds). On DL3DV and RealEstate10K, PPMem outperforms prior 3D-grounded world models in video quality while jointly producing metric-scale depth, surface normals, and exportable planar geometry, sustaining coherence across hundreds of frames under large viewpoint changes. Project page: https://hyzcluster.github.io/ppmem.
Tsinghua AI