Search papers, labs, and topics across Lattice.
The paper introduces Stereo-Inertial Poser, a real-time motion capture system that combines a single stereo camera with six IMUs to achieve metric-accurate and shape-aware 3D human motion estimation. It addresses the depth ambiguity inherent in monocular visual-inertial systems by using stereo vision for direct 3D keypoint extraction and body shape parameter estimation. A shape-aware fusion module dynamically harmonizes anthropometric variations with global translations, resulting in state-of-the-art performance at over 200 FPS without post-processing.
Achieve real-time, metric-accurate, and shape-aware 3D human motion capture using only a single stereo camera and six IMUs, outperforming existing monocular-based systems.
Recent advancements in visual-inertial motion capture systems have demonstrated the potential of combining monocular cameras with sparse inertial measurement units (IMUs) as cost-effective solutions, which effectively mitigate occlusion and drift issues inherent in single-modality systems. However, they are still limited by metric inaccuracies in global translations stemming from monocular depth ambiguity, and shape-agnostic local motion estimations that ignore anthropometric variations. We present Stereo-Inertial Poser, a real-time motion capture system that leverages a single stereo camera and six IMUs to estimate metric-accurate and shape-aware 3D human motion. By replacing the monocular RGB with stereo vision, our system resolves depth ambiguity through calibrated baseline geometry, enabling direct 3D keypoint extraction and body shape parameter estimation. IMU data and visual cues are fused for predicting drift-compensated joint positions and root movements, while a novel shape-aware fusion module dynamically harmonizes anthropometry variations with global translations. Our end-to-end pipeline achieves over 200 FPS without optimization-based post-processing, enabling real-time deployment. Quantitative evaluations across various datasets demonstrate state-of-the-art performance. Qualitative results show our method produces drift-free global translation under a long recording time and reduces foot-skating effects.
