Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the loss of plasticity in deep neural networks during continual learning and identifies dynamical isometry as a crucial factor in preserving this plasticity. By proposing an efficient isometry-promoting regularization scheme and introducing an adaptive optimizer, AdamO, the authors demonstrate that maintaining near-dynamical isometry allows networks to retain expressive capabilities while avoiding plasticity loss. Experimental results across various continual learning benchmarks show that their approach consistently matches or surpasses existing methods in mitigating plasticity degradation.
Dynamical isometry can be the key to preserving neural network plasticity in continual learning, enabling models to maintain performance without sacrificing expressiveness.
Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning. We revisit a class of networks that are almost-everywhere isometric while remaining universal Lipschitz function approximators, demonstrating that near-dynamical isometry is compatible with expressive nonlinear representations. For general architectures, we propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units. Building on this, we introduce AdamO, an Adam-style adaptive optimizer that decouples isometry regularization from gradient updates, analogous to AdamW. We further reinterpret prior plasticity-preserving approaches through the lens of dynamical isometry, showing that they target only a partial measure of isometry. Across supervised and reinforcement-learning continual-learning benchmarks designed to induce plasticity loss, our methods consistently match or outperform existing approaches.