Search papers, labs, and topics across Lattice.
This paper introduces Temporal Self-Imitation Learning (TSIL), a novel reinforcement learning framework that leverages temporally efficient trajectories to enhance policy learning in long-horizon robot manipulation tasks. By employing configuration-conditioned adaptive temporal targets and efficiency-weighted self-imitation, TSIL not only improves learning and task-completion efficiency but also ensures the preservation of successful behaviors throughout training. The approach was validated across 15 distinct manipulation tasks, demonstrating significant gains in efficiency and robustness compared to traditional methods reliant on reward shaping.
Temporal Self-Imitation Learning reveals that the structure of successful behaviors can serve as a powerful self-supervisory signal, drastically enhancing learning efficiency in complex tasks.
Long-horizon robot manipulation policies trained with reward shaping can still exploit dense rewards through inefficient interaction, while rare efficient behaviors may be forgotten during training. We argue that temporal efficiency itself provides a powerful and underutilized source of self-supervision for reinforcement learning. We introduce Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework that mines temporally efficient successful trajectories generated during learning and converts them into reusable supervision for future policy improvement. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviors through efficiency-weighted self-imitation learning. Across 15 distinct long-horizon manipulation tasks, TSIL consistently improves learning efficiency, task-completion efficiency, revisitation of fast successful behaviors, and robustness to unstable training conditions. More broadly, our results suggest that the temporal structure of successful behavior itself provides a scalable self-supervisory signal for reinforcement learning beyond manually engineered reward shaping alone.
