Search papers, labs, and topics across Lattice.
This paper introduces TacGen, a method that addresses the scarcity of tactile data by leveraging vision-to-touch alignment and generating tactile representations from RGB images. The authors demonstrate that their approach significantly enhances the accuracy of physical property estimations, such as mass and density, compared to vision-only methods, achieving substantial improvements in representation quality and manipulation capabilities. Key results show that the integration of tactile data not only boosts performance metrics but also preserves the majority of the gains when compared to traditional methods, highlighting the critical role of touch in physical-world representation learning.
Touch is not just an add-on; it fundamentally enhances object representation, leading to dramatic improvements in physical property estimation and manipulation tasks.
Touch resolves the physical-property ambiguity left by vision: exploratory contact recovers shape, texture, compliance, and material, and visuo-haptic object representations converge in ventral visual cortex. We ask whether representation learning can reproduce this grounding. TacGen mitigates the tactile-data scarcity bottleneck by combining pre-specified V+T contrastive alignment with a latent-space residual-MLP V->T generator that synthesizes tactile latents from RGB for tactile-data scaling. With matched DINOv2 backbones, splits, and probes, V+T improves matched V-only on mass (Delta R^2=+0.570), density (Delta acc=+0.067), hardness (+0.117), and uncertainty-banded force labels (Delta R^2=+0.281); all CIs exclude zero. The same representation lifts matched-capacity TACTO manipulation 0.246->0.979 while V-only capacity scaling accounts for only 4.5% of the gap, preserving 95.5%. The generator reaches cross-seed +0.589, with real tactile +0.585 inside the seed interval; the architecture comparison shows a 13pp downstream gap between reconstruction quality and representation utility. Across five-seed SSVTP/TVL reproductions, YCB-Sight transfer, three-backbone checks, permutation/random-feature controls, hash-verified manifests, and measured-force validation checks, the evidence supports the claim that touch supplies a necessary physical evidence channel for representations of contact-dependent properties.
CMU ML
Stanford HAI
UW