Search papers, labs, and topics across Lattice.
This paper introduces Stage-Transition Dense Reward (STDR), a novel framework that transforms unstructured expert videos into structured dense rewards for training reinforcement learning (RL) agents in robotic manipulation tasks. By leveraging semantic understanding to identify task stage structures, STDR provides two types of feedback during training: goal-directed stage-transition rewards and fine-grained within-stage progress signals. Experimental results across 14 manipulation tasks demonstrate that STDR significantly enhances sample efficiency and success rates compared to existing baselines, while also matching or exceeding the performance of handcrafted dense rewards in challenging scenarios.
Transforming unstructured expert videos into structured rewards, STDR boosts RL agent performance in robotic manipulation by improving both efficiency and robustness.
Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards, while manually designing dense shaping signals is costly and brittle to changes in environments and object configurations. This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch. STDR leverages semantic understanding to infer a task's stage structure from demonstrations, and delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage. Furthermore, an out-of-distribution (OOD) detection mechanism and a grasping regulation module are integrated to enhance robustness and prevent reward hacking. Experiments on 14 manipulation tasks across MetaWorld, ManiSkill, and Franka Kitchen show that STDR consistently improves sample efficiency and success rates over multiple baselines, and matches or surpasses handcrafted dense rewards on several challenging tasks. Real-robot evaluations further indicate that STDR assigns stable, progress-aligned rewards on successful executions while producing appropriately low rewards for failures, suggesting robustness to visual noise and better-calibrated reward assignment across settings.
