Search papers, labs, and topics across Lattice.
The paper introduces VGGT-Segmentor (VGGT-S), a framework for instance-level object segmentation across egocentric and exocentric views that addresses pixel-level projection drift in geometry-aware models. VGGT-S uses a novel Union Segmentation Head with mask prompt fusion, point-guided prediction, and iterative mask refinement to translate high-level feature alignment into precise segmentation masks. The method achieves state-of-the-art results on the Ego-Exo4D benchmark, with a correspondence-free self-supervised pretraining strategy that outperforms fully-supervised baselines.
Forget painstakingly annotating paired images: VGGT-Segmentor achieves state-of-the-art cross-view segmentation by cleverly pretraining on single images, sidestepping the need for correspondence.
Instance-level object segmentation across disparate egocentric and exocentric views is a fundamental challenge in visual understanding, critical for applications in embodied AI and remote collaboration. This task is exceptionally difficult due to severe changes in scale, perspective, and occlusion, which destabilize direct pixel-level matching. While recent geometry-aware models like VGGT provide a strong foundation for feature alignment, we find they often fail at dense prediction tasks due to significant pixel-level projection drift, even when their internal object-level textntion remains consistent. To bridge this gap, we introduce VGGT-Segmentor (VGGT-S), a framework that unifies robust geometric modeling with pixel-accurate semantic segmentation. VGGT-S leverages VGGT's powerful cross-view feature representation and introduces a novel Union Segmentation Head. This head operates in three stages: mask prompt fusion, point-guided prediction, and iterative mask refinement, effectively translating high-level feature alignment into a precise segmentation mask. Furthermore, we propose a single-image self-supervised training strategy that eliminates the need for paired annotations and enables strong generalization. On the Ego-Exo4D benchmark, VGGT-S sets a new state-of-the-art, achieving 67.7% and 68.0% average IoU for Ego to Exo and Exo to Ego tasks, respectively, significantly outperforming prior methods. Notably, our correspondence-free pretrained model surpasses most fully-supervised baselines, demonstrating the effectiveness and scalability of our approach.
