Search papers, labs, and topics across Lattice.
This paper introduces DeCAL, a novel vision-language-action model designed for dexterous manipulation that effectively integrates tactile sensing and physical dynamics. By employing a Mixture-of-Transformers architecture and Adaptive Visuo-Tactile Fusion, DeCAL enhances the model's ability to manage complex contact-rich interactions and dynamically regulate tactile inputs. Experimental results reveal that DeCAL achieves state-of-the-art performance with a 71% average success rate and 83.4% progress success rate across various tasks, showcasing its robustness in unseen scenarios.
DeCAL achieves a remarkable 71% success rate in dexterous manipulation tasks by seamlessly integrating tactile sensing with advanced vision-language-action modeling.
Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics. While recent works have incorporated tactile sensing into robotic manipulation, most approaches still rely on homogeneous multimodal fusion, lacking adaptive tactile integration and explicit modeling of physical dynamics. In this work, we present DeCAL, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation. Built upon a Mixture-of-Transformers (MoT) architecture, DeCAL leverages specialized experts for each capability while enabling efficient information flow among them. To effectively leverage tactile information, we introduce Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy. Furthermore, we propose Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics, equipping the policy with implicit physical world knowledge. Experimental results show that DeCAL consistently achieves state-of-the-art performance across all tasks, attaining a 71% average success rate and an 83.4% progress success rate, while also demonstrating strong generalization to unseen scenarios. The website is available at https://aureleopku.github.io/DeCAL.
