Search papers, labs, and topics across Lattice.
This paper introduces GeoLaV, a two-stage framework that enhances text-driven video segmentation by integrating 3D geometric knowledge into the training process. The first stage employs monocular geometry pretraining to develop geometry-consistent visual representations, while the second stage utilizes geometry-aware distillation to fine-tune the model on video datasets. Experimental results demonstrate that GeoLaV achieves state-of-the-art performance in referring video object segmentation, significantly improving spatiotemporal coherence and language grounding.
Geometry-aware distillation enables text-driven video segmentation models to achieve state-of-the-art performance by leveraging 3D geometric knowledge, even from 2D image datasets.
Text-driven Referring Video Object Segmentation (RVOS) aims to locate and segment target objects in videos given natural language. However, existing models are typically trained on 2D image or video datasets with naive segmentation losses, which overlooks the geometric consistency across frames and leads to weak spatial understanding. In this paper, we propose Geometry-enhanced Language-guided Video segmentation (GeoLaV), a two-stage framework that distills 3D geometric knowledge from images to enhance text-driven video segmentation. In the first stage, we perform monocular geometry pretraining with monocular novel-view synthesis, enabling the model to acquire geometry-consistent visual representations via spatial alignment on large-scale single-image datasets. In the second stage, we introduce geometry-aware distillation and fine-tune the model on video segmentation datasets, transferring 3D structural knowledge from a general 3D prior model. This process reinforces 3D awareness and improves both spatiotemporal coherence and language grounding in segmentation. Extensive experiments show that our method using only image segmentation data already provides notable zero-shot generalization in RVOS. When combined with geometry-aware distillation for fine-tuning on videos, our method achieves state-of-the-art performance across multiple RVOS benchmarks. The code is available at https://github.com/Tony1882880/GeoLaV.
