Search papers, labs, and topics across Lattice.
This paper introduces a generative learning-from-demonstration framework that leverages compositionality to model robotic behavior through object-centric neural fields, effectively linking perception and motion via shared representations. The approach employs a temporal mixture-of-experts (MoE) mechanism to generate complete motion trajectories from object-conditioned movement primitives, achieving significant data efficiency and systematic generalization across varied scene configurations. Experimental results show that the model excels in long-horizon manipulation tasks in both simulation and real-world settings, outperforming traditional image-based baselines while requiring less training data.
Achieving robust long-horizon manipulation with significantly less training data, this framework redefines efficiency in robotic learning through compositional motion generation.
Compositionality, by organizing complex behavior as combinations of simpler elements, enables robot learning that is scalable and data efficient. Leveraging this principle, we propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations. We render scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations, capturing positional and geometric variations in a smooth, consistent, and interpretable way. For motion generation, a temporal mixture-of-experts (MoE) employs a gating mechanism to combine object-conditioned movement primitives over time, producing complete trajectories. This spatial-temporal compositionality maintains the data efficiency of movement primitives while grounding motion in visual structure, enabling systematic generalization across diverse scene configurations. In simulation, long-horizon manipulation tasks are successfully completed using the proposed model, which requires significantly less training data than other image-based baselines. Real-world experiments further demonstrate the method's robustness to noise, its ability to generalize at the category level through language-based segmentation models, and its capacity to operate directly on 3D scene representations.
