Search papers, labs, and topics across Lattice.
This paper introduces DINO Eats CLIP (DEC), a framework for open-set 3D object retrieval that adapts the DINO self-supervised encoder to multi-view images. DEC employs a Chunking and Adapting Module (CAM) to dynamically integrate local view relations and a Virtual Feature Synthesis (VFS) module that leverages CLIP's vision-language space to synthesize features for unseen classes, mitigating bias towards known categories. Experiments on open-set 3DOR benchmarks demonstrate DEC's superior performance compared to CLIP-based methods.
DINO, not CLIP, might be the better foundation for open-set 3D object retrieval, especially when paired with dynamic view integration and virtual feature synthesis to avoid overfitting.
Vision foundation models have shown great promise for open-set 3D object retrieval (3DOR) through efficient adaptation to multi-view images. Leveraging semantically aligned latent space, previous work typically adapts the CLIP encoder to build view-based 3D descriptors. Despite CLIP's strong generalization ability, its lack of fine-grainedness prompted us to explore the potential of a more recent self-supervised encoder-DINO. To address this, we propose DINO Eats CLIP (DEC), a novel framework for dynamic multi-view integration that is regularized by synthesizing data for unseen classes. We first find that simply mean-pooling over view features from a frozen DINO backbone gives decent performance. Yet, further adaptation causes severe overfitting on average view patterns of known classes. To combat it, we then design a module named Chunking and Adapting Module (CAM). It segments multi-view images into chunks and dynamically integrates local view relations, yielding more robust features than the standard pooling strategy. Finally, we propose Virtual Feature Synthesis (VFS) module to mitigate bias towards known categories explicitly. Under the hood, VFS leverages CLIP's broad, pre-aligned vision-language space to synthesize virtual features for unseen classes. By exposing DEC to these virtual features, we greatly enhance its open-set discrimination capacity. Extensive experiments on standard open-set 3DOR benchmarks demonstrate its superior efficacy.
