Search papers, labs, and topics across Lattice.
This paper introduces VibeAct, a novel framework that integrates real vibrotactile sensing with simulation-based reinforcement learning to enhance dexterous manipulation in robots. By embedding piezoelectric microphones in a robot hand, the authors collect vibro-acoustic data that informs a tactile estimator for predicting contact and slip, facilitating the training of manipulation policies in simulation. The approach shows significant improvements in performance across various contact-rich tasks, particularly in scenarios requiring sustained reactive control, demonstrating effective transfer to a physical robot platform.
VibeAct reveals that leveraging real-time vibro-acoustic feedback can drastically enhance robotic dexterity in contact-rich environments, outperforming traditional proprioception methods.
Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones offer a compact and high-bandwidth way to sense these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully enough for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip. In the real world, we embed piezoelectric microphones into a dexterous robot hand and collect vibro-acoustic data through teleoperation, then replay the recordings in a calibrated digital clone to automatically label per-finger contact and slip. A tactile estimator learns to predict contact and slip from real microphone waveforms, while manipulation policies are trained in simulation on the same representation computed directly from simulated contacts. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across five contact-rich tasks spanning regrasping, in-hand reorientation, and insertion, VibeAct consistently outperforms a proprioception-and-point-cloud baseline in simulation, with the largest gains on tasks requiring sustained reactive control, where the continuous slip-magnitude channel proves the most informative observation. The learned policies transfer to a physical dexterous hand-arm platform, improving success rates on deployed tasks. Project videos and additional details are at https://vibeact.github.io/.
CMU ML