Search papers, labs, and topics across Lattice.
This paper addresses catastrophic forgetting in Continual Reinforcement Learning (CRL) by extending data rehearsal to value-based methods, specifically Deep Q-Networks. They identify limitations of actor-centric regularization and the lack of multi-cyclic environment evaluations in existing CRL research. The authors introduce Qreg+NWLU, which incorporates continuous data rehearsal with dynamic Q-value updates and "No-Wait" regularization, demonstrating improved learning efficiency, reduced forgetting, and enhanced knowledge transfer in multi-cyclic tasks.
Value function approximation in continual RL gets a boost: dynamically rehearsing Q-values and regularizing from the start significantly reduces catastrophic forgetting in multi-cyclic environments.
Data rehearsal has emerged as a leading approach for mitigating catastrophic forgetting in Continual Reinforcement Learning (CRL). However, existing work remains confined to policy gradient frameworks, regularizing only actors due to the performance degradation incurred by critic regularization. This actor-centric approach overlooks the potential of data rehearsal for value function approximation. Moreover, existing evaluations in CRL rarely consider multi-cyclic environments where task sequences repeat, a critical real-world scenario that exacerbates forgetting and plasticity. We investigate data rehearsal for Deep Q-Networks using Q-value regularization in multi-cyclic settings and propose Qreg+NWLU which introduces two simple modifications: (1) continuous data rehearsal that dynamically collects and updates stored Q-values throughout training, and (2) "No-Wait" regularization that applies immediately rather than after the first task. Together, these modifications yield improvements in learning efficiency, forgetting mitigation, and knowledge transfer over Qreg and conventional CRL methods within value function approximation settings.
