Search papers, labs, and topics across Lattice.
This paper analyzes Elastic Weight Consolidation (EWC) and identifies two key issues: gradient vanishing due to reliance on the Fisher Information Matrix (FIM) and redundant protection of irrelevant parameters. To address these, they introduce Logits Reversal (LR), which reverses logit values during FIM calculation to prevent both issues. Experiments demonstrate that EWC with LR (EWC-DR) significantly outperforms existing EWC variants across various continual learning tasks.
EWC, a classic method for continual learning, has been underperforming because it suffers from gradient vanishing and protects the wrong parameters – but a simple "Logits Reversal" trick fixes both.
Weight regularization methods in continual learning (CL) alleviate catastrophic forgetting by assessing and penalizing changes to important model weights. Elastic Weight Consolidation (EWC) is a foundational and widely used approach within this framework that estimates weight importance based on gradients. However, it has consistently shown suboptimal performance. In this paper, we conduct a systematic analysis of importance estimation in EWC from a gradient-based perspective. For the first time, we find that EWC's reliance on the Fisher Information Matrix (FIM) results in gradient vanishing and inaccurate importance estimation in certain scenarios. Our analysis also reveals that Memory Aware Synapses (MAS), a variant of EWC, imposes unnecessary constraints on parameters irrelevant to prior tasks, termed the redundant protection. Consequently, both EWC and its variants exhibit fundamental misalignments in estimating weight importance, leading to inferior performance. To tackle these issues, we propose the Logits Reversal (LR) operation, a simple yet effective modification that rectifies EWC's importance estimation. Specifically, reversing the logit values during the calculation of FIM can effectively prevent both gradient vanishing and redundant protection. Extensive experiments across various CL tasks and datasets show that the proposed method significantly outperforms existing EWC and its variants. Therefore, we refer to it as EWC Done Right (EWC-DR).
