Search papers, labs, and topics across Lattice.
DexViTac, a portable human-centric system, was developed to capture high-fidelity, multimodal (vision, tactile, kinematic) data for contact-rich dexterous manipulation in unstructured environments. A kinematics-grounded tactile representation learning algorithm was introduced to resolve semantic ambiguities in tactile signals. Experiments using a dataset of 2,400 demonstrations collected with DexViTac showed policies trained with the system achieved over 85% success on real-world manipulation tasks, outperforming baselines.
Policies trained on DexViTac's multimodal dataset achieve over 85% success in real-world dexterous manipulation, proving that high-fidelity tactile data unlocks a new level of robotic dexterity.
Large-scale, high-quality multimodal demonstrations are essential for robot learning of contact-rich dexterous manipulation. While human-centric data collection systems lower the barrier to scaling, they struggle to capture the tactile information during physical interactions. Motivated by this, we present DexViTac, a portable, human-centric data collection system tailored for contact-rich dexterous manipulation. The system enables the high-fidelity acquisition of first-person vision, high-density tactile sensing, end-effector poses, and hand kinematics within unstructured, in-the-wild environments. Building upon this hardware, we propose a kinematics-grounded tactile representation learning algorithm that effectively resolves semantic ambiguities within tactile signals. Leveraging the efficiency of DexViTac, we construct a multimodal dataset comprising over 2,400 visuo-tactile-kinematic demonstrations. Experiments demonstrate that DexViTac achieves a collection efficiency exceeding 248 demonstrations per hour and remains robust against complex visual occlusions. Real-world deployment confirms that policies trained with the proposed dataset and learning strategy achieve an average success rate exceeding 85% across four challenging tasks. This performance significantly outperforms baseline methods, thereby validating the substantial improvement the system provides for learning contact-rich dexterous manipulation. Project page: https://xitong-c.github.io/DexViTac/.
