Search papers, labs, and topics across Lattice.
The paper introduces SemanticFeels, an extension of NeuralFeels, which integrates semantic labeling with neural implicit shape representation using vision and touch for in-hand manipulation. This allows robots to perceive both the shape and material properties of objects, crucial for adaptive behavior in everyday tasks. The system uses a fine-tuned EfficientNet-B0 CNN to process high-resolution Digit tactile readings, generating local material predictions embedded into an augmented SDF network, achieving 79.87% material matching accuracy.
Robots can now "feel" material properties during in-hand manipulation with nearly 80% accuracy by fusing tactile sensing with neural implicit shape representations.
As robots become increasingly integrated into everyday tasks, their ability to perceive both the shape and properties of objects during in-hand manipulation becomes critical for adaptive and intelligent behavior. We present SemanticFeels, an extension of the NeuralFeels framework that integrates semantic labeling with neural implicit shape representation, from vision and touch. To illustrate its application, we focus on material classification: high-resolution Digit tactile readings are processed by a fine-tuned EfficientNet-B0 convolutional neural network (CNN) to generate local material predictions, which are then embedded into an augmented signed distance field (SDF) network that jointly predicts geometry and continuous material regions. Experimental results show that the system achieves a high correspondence between predicted and actual materials on both single- and multi-material objects, with an average matching accuracy of 79.87% across multiple manipulation trials on a multi-material object.
