Search papers, labs, and topics across Lattice.
This paper introduces MODIP, a novel framework that enhances the fine-tuning of diffusion policies (DPs) by integrating a world model to facilitate offline-to-online adaptation while maintaining the simplicity of behavioral cloning. By employing model predictive control (MPC) for trajectory generation and utilizing terminal state values for efficient planning, MODIP significantly reduces inference and training times. Experimental results demonstrate that MODIP surpasses behavioral cloning and competes effectively with existing diffusion policy reinforcement learning methods and strong model-based approaches on benchmark tasks.
MODIP outperforms traditional behavioral cloning and rivals advanced reinforcement learning methods, showcasing a new frontier in efficient policy fine-tuning for robot learning.
Diffusion policies (DPs) have emerged as expressive policy representations for robot learning, often used with imitation learning methods such as behavioral cloning (BC). However, while their success has largely been confined to BC, direct reinforcement learning (RL) fine-tuning remains challenging because actions are generated through a multi-step denoising process. In this work, we propose MODIP, a framework for the offline-to-online fine-tuning of DPs. Rather than directly applying RL to the DPs, MODIP leverages a world model (WM) to guide policy adaptation and keeps the simplicity and stability of BC. We utilize model predictive control (MPC) to generate high-quality trajectories within the WM, and use them as supervised targets for fine-tuning the DP. To make MPC planning efficient, MODIP uses a terminal state value instead of a policy-dependent state-action value, reducing inference time. Additionally, MODIP trains critics with policy-independent TD targets, reducing training time. Experiments on D4RL (MuJoCo, Kitchen) and RoboMimic tasks show that MODIP improves diffusion policies beyond BC, and is competitive with or outperforms diffusion policy RL fine-tuning methods and strong model-based baselines such as TD-MPC2.
