Search papers, labs, and topics across Lattice.
The paper introduces SpectralGCD, a novel approach to Generalized Category Discovery (GCD) that leverages CLIP-based cross-modal image-concept similarities to create a unified representation. SpectralGCD uses a large task-agnostic concept dictionary and spectral filtering based on a teacher model's cross-modal covariance matrix to select relevant concepts, thereby reducing reliance on visual cues and improving semantic quality. Experiments across six benchmarks demonstrate that SpectralGCD achieves comparable or superior accuracy to state-of-the-art methods with significantly reduced computational cost.
By distilling knowledge from a powerful teacher model using spectral filtering, SpectralGCD achieves state-of-the-art generalized category discovery performance at a fraction of the computational cost.
Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.
