Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the differential retention of knowledge in model-based reinforcement-learning agents, specifically within the DreamerV3 framework, revealing that while the world model retains crucial task information, the actor component suffers from significant forgetting. By employing graded dream rehearsal, the authors demonstrate that this method can effectively recover lost skills without additional environment interaction, achieving superior retention across multiple task sequences compared to traditional replay methods. The findings highlight that forgetting is primarily a channel issue rather than a memory issue, suggesting new avenues for enhancing continual learning in RL agents.
Forgetting in reinforcement learning agents is a channel problem, not a memory problem, and can be mitigated through graded dream rehearsal.
Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.