Search papers, labs, and topics across Lattice.
The paper introduces RoSHI, a wearable system combining sparse IMUs and Project Aria glasses for robust 3D human pose and body shape estimation in unconstrained environments. RoSHI leverages the complementary strengths of IMUs (occlusion robustness, high-speed motion capture) and egocentric SLAM (long-horizon consistency, upper body stabilization). Evaluation on a new dataset of agile activities shows RoSHI outperforms egocentric baselines and matches the performance of a state-of-the-art exocentric system, demonstrating its suitability for real-world humanoid policy learning.
Unlock robot learning from in-the-wild human demonstrations with RoSHI, a wearable system that rivals expensive motion capture setups for accuracy and robustness.
Scaling up robot learning will likely require human data containing rich and long-horizon interactions in the wild. Existing approaches for collecting such data trade off portability, robustness to occlusion, and global consistency. We introduce RoSHI, a hybrid wearable that fuses low-cost sparse IMUs with the Project Aria glasses to estimate the full 3D pose and body shape of the wearer in a metric global coordinate frame from egocentric perception. This system is motivated by the complementarity of the two sensors: IMUs provide robustness to occlusions and high-speed motions, while egocentric SLAM anchors long-horizon motion and stabilizes upper body pose. We collect a dataset of agile activities to evaluate RoSHI. On this dataset, we generally outperform other egocentric baselines and perform comparably to a state-of-the-art exocentric baseline (SAM3D). Finally, we demonstrate that the motion data recorded from our system are suitable for real-world humanoid policy learning. For videos, data and more, visit the project webpage: https://roshi-mocap.github.io/
