Search papers, labs, and topics across Lattice.
The paper introduces IGGT4D, a streaming instance-grounded geometry Transformer designed for online 4D scene understanding from continuous video streams. By employing causal spatial-temporal modeling to integrate historical context, IGGT4D achieves consistent geometry-instance representation in dynamic environments, overcoming limitations of existing methods that focus primarily on geometry. The authors validate their approach with a new large-scale dataset, InsScene4D-147K, and show that IGGT4D significantly outperforms current streaming baselines in tasks such as 3D reconstruction and instance tracking.
IGGT4D achieves unprecedented consistency in geometry-instance representation from streaming video, outperforming existing methods in dynamic scene understanding.
Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.
