Search papers, labs, and topics across Lattice.
This paper introduces SegDiff, a closed-loop visuomotor policy that enhances robot manipulation by integrating continuous trajectory prediction with keypose segmentation. By decomposing demonstrations into motion segments and employing a Dynamic Temporal Ensembling mechanism, SegDiff effectively addresses the limitations of both short-horizon continuous action sequences and discrete keypose methods. The results show significant performance improvements in both simulated and real-world scenarios, highlighting its capacity for long-horizon reasoning and real-time adaptability in dynamic environments.
SegDiff achieves real-time adaptability in robot manipulation by seamlessly combining continuous trajectory prediction with keypose segmentation, outperforming traditional methods.
Imitation learning enables robots to acquire manipulation skills from demonstrations by mapping observations to actions. Existing approaches predict either short-horizon continuous action sequences or discrete keyposes. However, continuous prediction methods suffer from compounding errors due to short prediction horizons and struggle with multi-modal action distributions, whereas keypose-based methods necessitate an external planner, constraining real-time applicability. To address these challenges, we introduce SegDiff, a closed-loop visuomotor policy that integrates the strengths of both paradigms. SegDiff decomposes demonstrations into motion segments between keyposes and learns to predict the continuous trajectory from the current state to the next keypose, enabling long-horizon prediction with real-time refinement. Furthermore, we leverage the capability of diffusion models and DDIM inversion to propose a Dynamic Temporal Ensembling mechanism, which allows the policy to efficiently respond to dynamic environments and mitigate discontinuities caused by inconsistent multi-modal sampling. SegDiff demonstrates significant performance gains over existing approaches across various simulated and real-world scenarios, indicating its strong ability to reason over extended temporal dependencies while maintaining real-time adaptability and control stability.
