Search papers, labs, and topics across Lattice.
OVS-DINO enhances open-vocabulary segmentation by addressing the attenuation of boundary awareness in DINO's internal representations. They introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to reactivate DINO's latent edge-sensitivity using structural priors from SAM and SAM-generated pseudo-masks for supervision. Experiments show OVS-DINO achieves state-of-the-art performance on multiple weakly-supervised OVS benchmarks, improving average score by 2.1% and significantly enhancing segmentation accuracy in complex scenarios.
Revitalizing the latent edge-sensitivity of DINO with SAM's structural priors yields state-of-the-art open-vocabulary segmentation, especially in cluttered scenes.
Open-Vocabulary Segmentation (OVS) aims to segment image regions beyond predefined category sets by leveraging semantic descriptions. While CLIP based approaches excel in semantic generalization, they frequently lack the fine-grained spatial awareness required for dense prediction. Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations. However, these methods still struggle with the precise edge perception necessary for high fidelity segmentation. In this paper, we analyze internal representations of DINO and discover that its inherent boundary awareness is not absent but rather undergoes progressive attenuation as features transition into deeper transformer blocks. To address this, we propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment with the Segment Anything Model (SAM). Specifically, we introduce a Structure-Aware Encoder (SAE) and a Structure-Modulated Decoder (SMD) to effectively activate boundary features of DINO using SAM's structural priors, complemented by a supervision strategy utilizing SAM generated pseudo-masks. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple weakly-supervised OVS benchmarks, improving the average score by 2.1% (from 44.8% to 46.9%). Notably, our approach significantly enhances segmentation accuracy in complex, cluttered scenarios, with a gain of 6.3% on Cityscapes (from 36.6% to 42.9%).
