Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces Value Disaggregation (VaDar), a novel approach for long-horizon reinforcement learning that trains a single, subgoal-independent policy by maximizing a linear combination of per-subgoal critic values. This method bypasses the need for explicit subgoal selection during deployment, a common bottleneck in traditional subgoal-conditioned policies. Experimental results demonstrate VaDar's competitiveness with, and frequent outperformance of, goal-conditioning baselines, notably achieving success on complex, non-sequential tasks in the RoboCasa suite where prior methods failed.
A single policy can master long-horizon tasks without explicit subgoal conditioning at inference, outperforming traditional methods by leveraging critic-level value disaggregation.
Long-horizon tasks are important for modern AI and robotics, yet pose a challenge for reinforcement learning. Historically, a popular approach has been to decompose a task into subgoals, and learn a subgoal-conditioned policy to execute each subtask in sequence. We introduce an alternative called Value Disaggregation (VaDar), which trains a single subgoal-independent policy to maximize a linear combination of per-subgoal critic values. Because subgoal information is needed only at training-time, VaDar removes the need for possibly costly or cumbersome subgoal-selection during policy deployment. Moreover, through careful experiments, we show that VaDar (1) is always competitive with, and frequently outperforms, natural baselines like goal-conditioning and (2) succeeds on tasks with non-sequential goal structure and multiple optimization objectives. In particular, VaDar is the first algorithm to solve challenging, long-horizon tasks in the RoboCasa suite, even when starting from a policy pretrained from behavior cloning with near-zero initial task success. Taken together, VaDar's successes suggest that subgoal information benefits actor-critic reinforcement learning by enhancing the efficacy of critic learning, whereas policy goal-conditioning is often unnecessary.
CMU ML