Search papers, labs, and topics across Lattice.
This paper introduces a topology-agnostic estimator that reconstructs the full mesh of deformable objects using only sparse touch data, without relying on visual input. The method employs a permutation-invariant cross-attention architecture capable of handling various object types, achieving a reduction in reconstruction error by approximately two-thirds compared to traditional methods. Additionally, the estimator utilizes deep-ensemble uncertainty to optimize touch locations, leading to further error reduction, particularly in scenarios with high self-occlusion.
Touch-based mesh reconstruction can achieve two-thirds lower error than traditional methods, even in complete darkness or heavy occlusion.
Estimating the full shape of a deformable object is especially challenging when vision is unavailable: in the dark, inside an opaque bag, behind the manipulating hand, or under heavy self-occlusion. Touch is the natural sensor in these settings, but touches are sparse and local. We present a single topology-agnostic estimator that reconstructs the full mesh of a deformable object from only a few touches and no vision, using one permutation-invariant cross-attention architecture that handles a 1D rope, a 2D cloth, and a 3D volumetric soft body. The learned estimator reduces reconstruction error by roughly two-thirds relative to non-learned geometric mesh completion and a Gaussian-process surface baseline, and it outperforms a simpler global-pool set encoder, with the gap growing as more touches are observed. We then show that the estimator's deep-ensemble uncertainty can be used to learn where to touch next, which lowers error further and beats both random touching and a Gaussian-process active baseline at sparse budgets. This gain is modest on average but grows with self-occlusion and on the error tail. When vision is also available, where to touch barely matters, motivating the vision-free setting we study.
