Search papers, labs, and topics across Lattice.
The GHOST framework introduces a hierarchical approach to robot manipulation by separating control into a high-level policy that predicts sub-goals from RGB-D observations and a low-level controller that executes actions based on these goals. This method enhances generalization beyond the training distribution and improves performance and robustness in various manipulation tasks compared to traditional flat policies. Notably, GHOST allows for easy integration of human demonstrations, facilitating the adaptation of learned skills to new objects and tasks with minimal additional data.
Hierarchical sub-goal policies enable robots to adapt to novel tasks with just a few human demonstrations, significantly enhancing their manipulation capabilities.
We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond the training distribution. GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations, and (ii) a low-level goal-conditioned controller that executes embodiment-specific actions. To condition image-based policies on 3D goals, we introduce a simple spatial interface that projects predicted goals into the image plane and represents them as end-effector heatmaps. Across a suite of manipulation tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy. Further, we show that this hierarchical interface also makes it easy to incorporate human demonstrations without relying on (noisy) action retargeting. As sub-goals are largely embodiment-agnostic, we train the high-level policy on human video to specify how learned skills should be applied and composed, while keeping the low-level policy trained purely on robot data. This hierarchy enables adaptation to novel objects and task variations using a small number of human demonstrations.
CMU ML