Search papers, labs, and topics across Lattice.
The paper investigates the application of weight regularization, specifically Elastic Weight Consolidation (EWC), within parameter-efficient continual learning (PECL) using low-rank adapters (LoRA). They propose EWC-LoRA, which regularizes the shared low-rank update using EWC to mitigate task interference while maintaining constant storage and inference costs. Experiments on several benchmarks demonstrate that EWC-LoRA achieves a better stability-plasticity trade-off compared to existing low-rank CL methods, highlighting the effectiveness of weight regularization even with low-rank parameterizations.
Weight regularization, often overlooked in parameter-efficient continual learning, can still significantly improve the stability-plasticity trade-off, even when using low-rank adapters.
Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practical, computational- and memory-efficient solution for CL with PTMs, and provides insights that may inform the broader application of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving a stability-plasticity trade-off superior to existing low-rank CL approaches. These results indicate that, even under low-rank parameterizations, weight regularization remains an effective mechanism for mitigating task interference. Code is available at: https://github.com/yaoyz96/low-rank-cl.
