Search papers, labs, and topics across Lattice.
This paper introduces Concept-Constrained Prompt Learning (CCPL), a novel framework designed to enhance few-shot adaptation of CLIP by anchoring learnable class prompts to frozen concept-level text prototypes. By employing a text-space cosine consistency objective and concept dropout, CCPL mitigates overfitting to base-class supervision and improves transferability to unseen classes. Experimental results demonstrate that CCPL achieves significant performance gains on DTD and EuroSAT datasets while maintaining stability on OxfordPets, indicating that concept constraints can effectively align with dataset semantics for better classification outcomes.
CCPL boosts few-shot CLIP adaptation performance by anchoring prompts to frozen concept prototypes, revealing that concept constraints can enhance transferability to unseen classes.
Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization framework that anchors learnable class prompts to frozen concept-level text prototypes without updating CLIP encoders. CCPL learns a set of shared context tokens, instantiates class prompts by appending class names, and constructs frozen concept prototypes from a class-level concept bank. During training, a text-space cosine consistency objective aligns learnable class-prompt embeddings with frozen concept prototypes; concept dropout provides additional regularization against over-reliance on fixed concept lists. At inference, CCPL optionally fuses class-prompt logits with concept-prototype logits using a controllable ensemble weight alpha. Our default configuration uses text-space concept regularization lambda = 0.5, concept dropout p = 0.3 and weak concept-guided fusion (alpha = 0.1), with no KL-based prediction consistency term. Experiments under identical automatically-generated fallback splits show that CCPL improves the base-to-new harmonic mean on DTD (+0.6) and EuroSAT (+2.9) compared with CoOp, while remaining near-neutral on OxfordPets (-0.1). Ablations indicate that text-space concept regularization is consistently beneficial, while the best concept-guided inference strength is dataset- and protocol-sensitive. These results suggest concept constraints are most effective when concept prototypes align naturally with dataset semantics, and identify fine-grained categories as a current boundary condition. The code is released at: https://github.com/richael-sang/concept-constrained-prompt-learning.
