Search papers, labs, and topics across Lattice.
The paper introduces DreamMimic, a framework for vision-based whole-body loco-manipulation in humanoid robots that leverages world-model-assisted policy distillation. By repurposing a Dreamer-style RSSM to learn predictive latent dynamics, the method enhances the representation space and provides multi-step supervision, effectively reducing long-term drift in visual inputs. Experiments demonstrate that DreamMimic outperforms existing vision-based baselines in loco-manipulation tasks, highlighting the potential of world models to stabilize policy learning in complex, contact-rich environments.
World models can significantly enhance the stability and performance of humanoid robots in complex loco-manipulation tasks, outperforming traditional methods.
Vision-based whole-body loco-manipulation on humanoid robots is challenging due to partial observability, contact-rich dynamics, and the difficulty of learning long-horizon behaviors from high-dimensional visual inputs. We present \href{https://github.com/DreamMimic/DreamMimic}{DreamMimic}, a framework that distills privileged teacher policies into vision-based humanoid controllers via world-model-assisted distillation. Instead of using a Dreamer-style RSSM for planning, we repurpose it to learn predictive latent dynamics that serve as both a representation space and an action-conditioned multi-step supervision signal, while exposing compact predictive features to the student policy to reduce long-term drift. Beyond standard reconstruction objectives for proprioceptive and visual observations, we add auxiliary prediction heads for privileged state, contact, object state, and reward estimation. These heads provide additional supervision related to agent--object interaction and task progress, encouraging the latent representation to retain signals that are useful for contact-rich loco-manipulation. We further introduce Performance-Conditioned Guidance (PCG), a reward-driven adaptive distillation schedule that computes performance scores for both teacher and student to dynamically balance guidance and exploration. PCG prevents both premature teacher annealing and excessive teacher interference in challenging visual settings. Experiments on OMOMO and BEHAVE show improved tracking-based loco-manipulation performance over strong vision-based baselines, without exposing online privileged interaction states to the student at deployment. Qualitative simulations further examine morphology and simulator changes. These results suggest that world models can provide a useful mechanism for stabilizing visual policy distillation in contact-rich humanoid behaviors.
