Search papers, labs, and topics across Lattice.
This paper benchmarks different vision-torque fusion strategies within diffusion-based manipulation policies for contact-rich tasks. It introduces an adaptive integration strategy that selectively uses F/T signals only during contact phases, determined by a learned gating mechanism. Experiments show this adaptive approach outperforms existing fusion methods, improving success rates by 14%.
Contact-rich robotic manipulation gets a 14% performance boost by adaptively ignoring force/torque signals when they're irrelevant, and fusing them intelligently with vision when contact matters.
Vision-based policies have achieved a good performance in robotic manipulation due to the accessibility and richness of visual observations. However, purely visual sensing becomes insufficient in contact-rich and force-sensitive tasks where force/torque (F/T) signals provide critical information about contact dynamics, alignment, and interaction quality. Although various strategies have been proposed to integrate vision and F/T signals, including auxiliary prediction objectives, mixture-of-experts architectures, and contact-aware gating mechanisms, a comparison of these approaches remains lacking. In this work, we provide a comparison study of different F/T-vision integration strategies within diffusion-based manipulation policies. In addition, we propose an adaptive integration strategy that ignores F/T signals during non-contact phases while adaptively leveraging both vision and torque information during contact. Experimental results demonstrate that our method outperforms the strongest baseline by 14% in success rate, highlighting the importance of contact-aware multimodal fusion for robotic manipulation.
