Search papers, labs, and topics across Lattice.
This paper introduces the Lifelong Mixture of Dynamic Experts (LiMoDE), a two-stage learning framework designed to enhance robot manipulation by effectively leveraging prior knowledge for continuous task adaptation. By employing a dynamic mixture of experts (MoE) structure during multi-task pre-training, the approach activates a variable number of specialized experts based on motion information to handle diverse short-term manipulations. The results show that LiMoDE achieves superior performance and robust lifelong adaptation with minimal additional parameters and inference overhead, addressing the limitations of previous methods in skill reuse and interaction modeling.
LiMoDE enables robots to adaptively combine learned skills for new tasks without catastrophic forgetting, significantly enhancing their manipulation capabilities.
Building a generalist robot that can leverage prior knowledge for continuous task adaptation remains a significant challenge. Previous works alleviate the catastrophic forgetting problem by parameter-efficient fine-tuning for single-task adaptation. However, they fail to extract reusable skills and model the interaction with other skills effectively. Recent works try to address these issues by learning prompts. Differently, this paper presents an architectural perspective on the Lifelong Mixture of Dynamic Experts (\textit{LiMoDE}), a novel two-stage learning scheme for lifelong robot manipulation. Specifically, a dynamic MoE structure is first proposed in the multi-task pre-training stage to learn prior knowledge, where a varied number of heterogeneous experts are activated based on the motion information to address different short-term manipulations. Subsequently, in the task adaptation stage, we design a lifelong MoE adaptation mechanism % (LiMoEAM) that learns lifelong experts and dynamically combines them with frozen ones for new tasks, facilitating the knowledge transfer during adaptation. The proposed \textit{LiMoDE} is evaluated on both the simulated lifelong learning benchmark and real-world tasks. Extensive experiments demonstrate its effectiveness in achieving superior performance and strong lifelong adaptation by introducing a moderate number of additional trainable parameters and inference overhead.
