Search papers, labs, and topics across Lattice.
This paper introduces Knowledge-Enhanced Clustering (KEC), a novel image clustering method that leverages hierarchical concept-attribute knowledge extracted from large language models (LLMs). KEC uses structured prompts to condense textual labels into abstract concepts and extract discriminative attributes for both individual concepts and similar concept pairs. By instantiating this knowledge for each image and combining it with visual features, KEC achieves significant and consistent improvements across 20 diverse datasets compared to existing methods using textual knowledge.
LLMs can automatically generate hierarchical concept-attribute knowledge that significantly boosts image clustering performance, even surpassing zero-shot CLIP in many cases.
Image clustering aims to group images in an unsupervised fashion. Traditional methods focus on knowledge from visual space, making it difficult to distinguish between visually similar but semantically different classes. Recent advances in vision-language models enable the use of textual knowledge to enhance image clustering. However, most existing methods rely on coarse class labels or simple nouns, overlooking the rich conceptual and attribute-level semantics embedded in textual space. In this paper, we propose a knowledge-enhanced clustering (KEC) method that constructs a hierarchical concept-attribute structured knowledge with the help of large language models (LLMs) to guide clustering. Specifically, we first condense redundant textual labels into abstract concepts and then automatically extract discriminative attributes for each single concept and similar concept pairs, via structured prompts to LLMs. This knowledge is instantiated for each input image to achieve the knowledge-enhanced features. The knowledge-enhanced features with original visual features are adapted to various downstream clustering algorithms. We evaluate KEC on 20 diverse datasets, showing consistent improvements across existing methods using additional textual knowledge. KEC without training outperforms zero-shot CLIP on 14 out of 20 datasets. Furthermore, the naive use of textual knowledge may harm clustering performance, while KEC provides both accuracy and robustness.
