Search papers, labs, and topics across Lattice.
This paper introduces the Coupled Comprehensive Generative Replay (C^2GR) framework, designed to address the problem of catastrophic forgetting in universal segmentation models during task-incremental learning. By synthesizing image-mask pairs from previous tasks while managing concurrent shifts in appearance and segmentation objectives, C^2GR leverages a Bayesian Joint Diffusion method for realistic image generation and a Relation-aware Unified Prompt Synchronization scheme for optimizing both the generator and segmentor. Experimental results across 20 diverse tasks show that C^2GR achieves only a 2.44% performance drop compared to joint training, significantly mitigating forgetting in continual learning scenarios.
Forgetting in universal segmentation models can be reduced to just 2.44% with a novel generative replay framework that synchronizes task relations.
Universal segmentation models exhibit significant potential for diverse tasks involving different imaging modalities and segmentation objectives. Task-Incremental Learning provides a privacy-preserving approach to continually evolve a universal model on tasks from sequentially-arriving medical departments. However, training the model solely on the incoming task induces forgetting on past tasks, since consecutive tasks exhibit concurrent shifts in image appearance and segmentation objective. To address this problem, we propose a novel Coupled Comprehensive Generative Replay (C^2GR) framework that simultaneously synthesizes image-mask pairs of previous tasks to mitigate forgetting under concurrent appearance and objective shifts. This requires preserving image-mask correspondence for structure-realistic generation and bridging asynchronous optimization of the generator and segmentor for segmentation-oriented generation. Specifically, we propose a Bayesian Joint Diffusion (BJD) method that formulates the correspondence as conditional distributions optimized via conditional denoising. Furthermore, we develop a Relation-aware Unified Prompt Synchronization (RUPS) scheme to simultaneously modulate the generator and segmentor via a shared task-relation-aware prompt for synchronizing their optimization. Experiments on 20 tasks spanning diverse modalities and objectives demonstrate that C^2GR exhibits only a 2.44% drop in overall performance compared to joint training with all task data, effectively alleviating forgetting from the concurrent shifts. Our code will be made publicly available at https://github.com/mar-cry/C2GR.
AI2