Search papers, labs, and topics across Lattice.
H-Flow is introduced, a self-supervised method for estimating dense human scene flow from monocular video by jointly predicting pose and depth using a multi-head transformer. The approach overcomes the lack of direct supervision by encoding geometric, structural, and biomechanical priors as cross-modal training objectives, effectively anchoring the network in the physics of human motion. Experiments show H-Flow outperforms existing scene-flow and parametric baselines and generalizes to in-the-wild videos, validated using a new high-fidelity synthetic benchmark, DynAct4D, with dense flow annotations.
Physics-inspired self-supervision unlocks accurate dense human scene flow estimation from monocular video, even without direct flow labels.
Parametric human models capture global pose but cannot represent the non-rigid surface dynamics of clothing and soft tissue. Generic scene flow estimates dense motion but breaks down on articulated bodies, where pixel-level supervision is also intractable to acquire. We introduce H-Flow, a dense human scene flow that captures both skeletal kinematics and surface deformation. A unified multi-head transformer estimates flow from monocular video, jointly predicting pose and depth as companion outputs. The challenge lies in the lack of supervision. In place of unattainable labels, we anchor the network in the physics of human motion, encoding geometric, structural, and biomechanical priors as cross-modal training objectives. We further introduce DynAct4D, a high-fidelity synthetic benchmark providing dense flow annotations across diverse subjects, garments, and motions. On standard benchmarks, H-Flow outperforms scene-flow and parametric baselines, and generalizes zero-shot to in-the-wild video. Code, models, and the DynAct4D benchmark will be released upon publication
