Search papers, labs, and topics across Lattice.
SpaCeFormer introduces a fast, proposal-free transformer architecture for open-vocabulary 3D instance segmentation, addressing the speed bottlenecks of previous multi-stage pipelines and the limitations of pseudo-labeled end-to-end approaches. The model leverages spatial window attention with Morton-curve serialization and a RoPE-enhanced decoder to directly predict instance masks. Evaluated on ScanNet200, ScanNet++, and Replica datasets, SpaCeFormer achieves state-of-the-art performance, demonstrating a 2.8x improvement in zero-shot mAP compared to prior proposal-free methods and surpassing all existing methods in ScanNet++ and Replica.
Open-vocabulary 3D instance segmentation just got 100x faster, thanks to a new transformer architecture that ditches region proposals and fragmented masks.
Open-vocabulary 3D instance segmentation is a core capability for robotics and AR/VR, but prior methods trade one bottleneck for another: multi-stage 2D+3D pipelines aggregate foundation-model outputs at hundreds of seconds per scene, while pseudo-labeled end-to-end approaches rely on fragmented masks and external region proposals. We present SpaCeFormer, a proposal-free space-curve transformer that runs at 0.14 seconds per scene, 2-3 orders of magnitude faster than multi-stage 2D+3D pipelines. We pair it with SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset (3.0M multi-view-consistent captions over 604K instances from 7.4K scenes) built through multi-view mask clustering and multi-view VLM captioning; it reaches 21x higher mask recall than prior single-view pipelines (54.3% vs 2.5% at IoU>0.5). SpaCeFormer combines spatial window attention with Morton-curve serialization for spatially coherent features, and uses a RoPE-enhanced decoder to predict instance masks directly from learned queries without external proposals. On ScanNet200 we achieve 11.1 zero-shot mAP, a 2.8x improvement over the prior best proposal-free method; on ScanNet++ and Replica, we reach 22.9 and 24.1 mAP, surpassing all prior methods including those using multi-view 2D inputs.
NVIDIA