Search papers, labs, and topics across Lattice.
This paper introduces KineFuse, a novel approach for tracking the 6D pose of occluded objects during dexterous in-hand manipulation by leveraging sparse haptic signals from multi-fingered hands. By employing a kinematic-aware finger-level encoder, the authors demonstrate that their method significantly enhances pose tracking performance, especially in sequential scenarios, compared to alternative designs. Key findings reveal that the structured encoder effectively utilizes visual information for translation while relying on haptic data for rotation, leading to improved tracking accuracy and success in downstream tasks.
Occluded-object pose tracking can be revolutionized by a kinematic-aware approach that smartly fuses haptic and visual signals, achieving up to 15 times better performance in real-world manipulation tasks.
Dexterous in-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera. We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact, to complement a pretrained visual pose tracker under occlusion. We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation: per-frame refinement, sequential open-loop tracking, and closed-loop manipulation. Our experiments reveal that (i) per-frame evaluation cannot distinguish encoder quality, while sequential tracking amplifies architectural differences by up to 15 times; (ii) the structured encoder learns task-specific cross-modal gating, using vision exclusively for translation and dedicating one attention head to haptics for rotation, without explicit supervision; and (iii) compact finger-level tokenization with 4 tokens outperforms both flat fusion and joint-level representations, which suppress vision through norm dominance. We validate that improved tracking yields higher success in a downstream reorientation task and provide qualitative real-world demonstrations. Our project page is available at https://cold-young.github.io/kine-fuse/.
