Search papers, labs, and topics across Lattice.
The paper introduces SAILS, a training-free framework for Class-Incremental Semantic Segmentation (CISS) that addresses limitations of continual learning such as repeated retraining and forgetting. SAILS leverages the Segment Anything Model (SAM) for zero-shot region extraction and associates these regions with semantic prototypes in a fixed feature space, incorporating selective intra-class clustering for improved representation. Experiments demonstrate that SAILS outperforms training-based CISS methods, especially in long task sequences, while eliminating forgetting and exhibiting positive backward transfer due to its training-free nature.
Forget retraining? This new method uses SAM to segment anything and then incrementally learns semantics, beating standard continual learning approaches *without any training*.
Continual learning remains constrained by the need for repeated retraining, high computational costs, and the persistent challenge of forgetting. These factors significantly limit the applicability of continuous learning in real-world settings, as iterative model updates require significant computational resources and inherently exacerbate forgetting. We present SAILS -- Segment Anything with Incrementally Learned Semantics, a training-free framework for Class-Incremental Semantic Segmentation (CISS) that sidesteps these challenges entirely. SAILS leverages foundational models to decouple CISS into two stages: Zero-shot region extraction using Segment Anything Model (SAM), followed by semantic association through prototypes in a fixed feature space. SAILS incorporates selective intra-class clustering, resulting in multiple prototypes per class to better model intra-class variability. Our results demonstrate that, despite requiring no incremental training, SAILS typically surpasses the performance of existing training-based approaches on standard CISS datasets, particularly in long and challenging task sequences where forgetting tends to be most severe. By avoiding parameter updates, SAILS completely eliminates forgetting and maintains consistent, task-invariant performance. Furthermore, SAILS exhibits positive backward transfer, where the introduction of new classes can enhance performance on previous classes.
