Search papers, labs, and topics across Lattice.
This paper introduces Prototype-guided Text Prompt Selection (ProTPS), a novel continual learning approach that learns class-specific vision prototypes to guide the selection and learning of unique text prompts. By increasing training flexibility, ProTPS encourages the learning of distinct text prompts for new classes, mitigating catastrophic forgetting. Experiments across class-incremental, cross-dataset continual, and a new real-world marine species dataset (Marine112) demonstrate that ProTPS achieves state-of-the-art performance.
Forget catastrophic forgetting: ProTPS leverages vision prototypes to guide text prompt learning, achieving near-upper-bound performance in continual learning scenarios.
For continual learning, text-prompt-based methods leverage text encoders and learnable prompts to encode semantic features for sequentially arrived classes over time. A common challenge encountered by existing works is how to learn unique text prompts, which implicitly carry semantic information of new classes, so that the semantic features of newly arrived classes do not overlap with those of trained classes, thereby mitigating the catastrophic forgetting problem. To address this challenge, we propose a novel approach Prototype-guided Text Prompt Selection (ProTPS)'' to intentionally increase the training flexibility thus encouraging the learning of unique text prompts. Specifically, our ProTPS learns class-specific vision prototypes and text prompts. Vision prototypes guide the selection and learning of text prompts for each class. We first evaluate our ProTPS in both class incremental (CI) setting and cross-datasets continual (CDC) learning setting. Because our ProTPS achieves performance close to the upper bounds, we further collect a real-world dataset with 112 marine species collected over a span of six years, named Marine112, to bring new challenges to the community. Marine112 is authentically suited for the class and domain incremental (CDI) learning setting and is under natural long-tail distribution. The results under three settings show that our ProTPS performs favorably against the recent state-of-the-art methods. The implementation code and Marine112 dataset will be released upon the acceptance of our paper.
UW