Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Calibrated Partial Resets (CPR), an innovative optimizer designed to prevent policy collapse in continual reinforcement learning by selectively reinitializing low-utility neurons without destabilizing training. By scaling the pull strength of neuron resets based on their utility, CPR maintains gradient flow and enhances plasticity while avoiding the pitfalls of full reinitialization and uniform decay methods. The results show that CPR effectively prevents policy collapse over extensive training and outperforms existing techniques on multiple benchmarks, establishing a new approach to managing neuron utility in continual learning scenarios.
CPR uniquely balances neuron reinitialization and training stability, preventing policy collapse while enhancing performance across challenging continual learning tasks.
Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning. Recently, neuron resets have been used to maintain gradient flow and restore plasticity. However, full unit reinitialization often sacrifices peak performance and can destabilize training, leading to policy collapse. To preserve plasticity without destabilizing training, we propose Calibrated Partial Resets (CPR), an optimizer that periodically pulls low-utility neurons toward their initialization, with pull strength scaled by each neuron's utility. Unlike binary reset methods, partial resets avoid brittleness; unlike uniform decay, calibrated utility-scaling concentrates adjustment on the units that need it most. Among compared methods, only CPR avoids policy collapse over 400M training steps in SlipperyAnt, and it outperforms prior decay and reset-based methods on Continual MetaWorld and Continual MinAtar benchmarks. Ablations reveal a tunable trade-off between plasticity and peak performance, highlighting utility-scaled reinitialization as a promising direction for continual learning.