Search papers, labs, and topics across Lattice.
The paper introduces DeCLIP, a framework to improve the performance of CLIP-based models on open-vocabulary dense prediction tasks by addressing limitations in local feature representation. DeCLIP decouples the self-attention module in CLIP to generate content and context features, enhancing the latter with semantic correlations from Vision Foundation Models (VFMs) and object integrity cues from diffusion models, and the former with image crop representations and region correlations from VFMs. Experiments across 2D detection/segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation demonstrate state-of-the-art performance, showing DeCLIP's effectiveness in open-vocabulary dense perception.
CLIP's image tokens struggle to aggregate information from spatially or semantically related regions, but DeCLIP fixes this by decoupling self-attention and distilling knowledge from VFMs and diffusion models.
Dense visual perception tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense perception often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain ``content''and ``context''features respectively. \revise{The context features are enhanced by jointly distilling semantic correlations from Vision Foundation Models (VFMs) and object integrity cues from diffusion models, thereby enhancing spatial consistency. In parallel, the content features are aligned with image crop representations and constrained by region correlations from VFMs to improve local discriminability. Extensive experiments demonstrate that DeCLIP establishes a solid foundation for open-vocabulary dense perception, consistently achieving state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.} Code is available at https://github.com/xiaomoguhz/DeCLIP
