Search papers, labs, and topics across Lattice.
The paper introduces {\tau}, a touch-augmented Vision-Language-Action (VLA) framework that learns action-conditioned spatiotemporal tactile representations using future visual supervision, addressing the challenges of limited task-specific demonstrations and high-dimensional tactile signals. By leveraging the Joint-Embedding Predictive Architecture (JEPA) for latent space supervision, {\tau} effectively integrates tactile information with vision-language features, enhancing action generation capabilities. Experimental results demonstrate that {\tau} surpasses existing models in manipulation tasks, achieving better generalization to unseen objects and scenes, thus improving overall manipulation performance and robustness.
Future visual supervision enables touch-augmented VLA models to significantly enhance manipulation performance, even with limited data.
Learning the informative tactile representation while effectively adapting it to pretrained Vision-Language-Action (VLA) models remains challenging at both the data and modeling levels. At the data level, limited task-specific demonstrations constrain representation quality, whereas large-scale pretraining incurs substantial costs. At the modeling level, existing methods either focus on instantaneous contact states or model temporal interaction dynamics using 6D wrench sequences, leaving high-dimensional tactile signals underexplored. To address these challenges, we present {\tau}, a touch-augmented VLA framework that learns an action-conditioned spatiotemporal tactile representation from future visual supervision inspired by the Joint-Embedding Predictive Architecture (JEPA), and fuses it with vision-language features for action generation under limited data. This supervision operates in latent space and is used only during training, adding no deployment overhead. We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks. Experiments show that {\tau} outperforms existing models and generalizes to unseen objects and scenes, delivering improved manipulation performance and robustness
