Search papers, labs, and topics across Lattice.
This paper introduces E-3DPSM, an event-driven continuous pose state machine designed for event-based egocentric 3D human pose estimation using head-mounted devices. E-3DPSM aligns continuous human motion with fine-grained event dynamics by evolving latent states and predicting continuous changes in 3D joint positions based on observed events, which are then fused with direct 3D pose predictions. The method achieves state-of-the-art results on two benchmarks, improving accuracy by up to 19% (MPJPE) and temporal stability by up to 2.7x, while running in real-time.
Event cameras can now enable significantly more accurate and stable egocentric 3D human pose estimation, thanks to a novel state machine approach that directly leverages fine-grained event dynamics.
Event cameras offer multiple advantages in monocular egocentric 3D human pose estimation from head-mounted devices, such as millisecond temporal resolution, high dynamic range, and negligible motion blur. Existing methods effectively leverage these properties, but suffer from low 3D estimation accuracy, insufficient in many applications (e.g., immersive VR/AR). This is due to the design not being fully tailored towards event streams (e.g., their asynchronous and continuous nature), leading to high sensitivity to self-occlusions and temporal jitter in the estimates. This paper rethinks the setting and introduces E-3DPSM, an event-driven continuous pose state machine for event-based egocentric 3D human pose estimation. E-3DPSM aligns continuous human motion with fine-grained event dynamics; it evolves latent states and predicts continuous changes in 3D joint positions associated with observed events, which are fused with direct 3D human pose predictions, leading to stable and drift-free final 3D pose reconstructions. E-3DPSM runs in real-time at 80 Hz on a single workstation and sets a new state of the art in experiments on two benchmarks, improving accuracy by up to 19% (MPJPE) and temporal stability by up to 2.7x. See our project page for the source code and trained models.
