Search papers, labs, and topics across Lattice.
This paper introduces Recurrent Generative Replay (REGEN), a continual imitation learning framework that leverages World Action Models (WAMs) to generate pseudo-replay trajectories, allowing robots to rehearse previously learned tasks without needing to store original human demonstrations. By recursively querying the WAM, REGEN synthesizes these trajectories based solely on task instructions and current observations, significantly mitigating catastrophic forgetting by up to 50% compared to traditional sequential fine-tuning. The findings highlight the potential of WAMs in enabling continual learning in robotics while approaching the performance of methods reliant on real replay data, despite challenges like visual degradation and action-observation inconsistency.
REGEN enables robots to continually learn and rehearse tasks without the need for storing human demonstrations, cutting catastrophic forgetting by 50%.
Going beyond predicting robot actions, World Action Models (WAMs) can also generate future visual observations. We build on this generative capability to propose Recurrent Generative Replay (REGEN), a continual imitation learning framework that synthesizes pseudo-replay trajectories, enabling a robot policy to rehearse previously learned tasks without storing their original human demonstrations. During continual adaptation, REGEN recursively queries the WAM to synthesize pseudo-replay trajectories conditioned only on prior task instructions and current-task observations. Experiments in both simulation and real-world manipulation settings show that REGEN reduces catastrophic forgetting by up to $50\%$ relative to sequential fine-tuning, while approaching the performance of privileged experience replay methods that require access to real replay data. Finally, we analyze the factors limiting generated replay, identifying long-horizon visual degradation and action-observation inconsistency as the primary bottlenecks. Our results establish WAMs as a promising foundation for continual robot learning without stored demonstrations.
