Search papers, labs, and topics across Lattice.
This paper introduces UniTac, the first unified multimodal model specifically designed for tactile understanding and generation, addressing the lack of research in this area. By modeling tactile interactions as a transition from non-contact to contact, UniTac captures both sensor and object attributes through a dual-level representation. The model achieves state-of-the-art performance in tactile understanding tasks and generates realistic tactile signals, demonstrating its effectiveness in integrating physical interactions across diverse sensors.
UniTac achieves state-of-the-art tactile understanding and generation by seamlessly integrating sensor and object attributes, revolutionizing how we interpret touch across modalities.
Unified multimodal models (UMMs) have shown great promise in integrating understanding and generation across diverse modalities. However, existing research rarely extends this paradigm to the tactile domain, where both object-level semantics and sensor-level configurations jointly determine the meaning of touch. To address this gap, we propose UniTac, the first UMM designed for tactile understanding and generation. UniTac models the tactile process as a transition from non-contact to contact, capturing the physical interaction between sensors and objects through a dual-level representation that encodes both sensor and object attributes. For tactile understanding, UniTac introduces two tasks, object property description and sensor identification, to enhance reasoning over physical and cross-sensor information. For tactile generation, we design a two-stage training paradigm consisting of reconstruction and alignment, together with a sensor-prior-based sampling strategy that simulates realistic tactile contact. Trained on large-scale multi-sensor datasets, UniTac achieves state-of-the-art performance in tactile understanding and generates realistic tactile signals across sensors.
