Search papers, labs, and topics across Lattice.
CHROMM is introduced as a unified framework for jointly estimating cameras, scene point clouds, and human meshes from multi-person multi-view videos, eliminating the need for external modules or preprocessing. It integrates geometric and human priors from Pi3X and Multi-HMR within a single trainable neural network, incorporating a scale adjustment module and a multi-view fusion strategy. The framework also employs a geometry-based multi-person association method, demonstrating competitive performance and achieving over 8x speedup compared to optimization-based multi-view approaches on datasets like EMDB and EgoExo4D.
Reconstructing humans and their environments from multi-view video can now be done in a single pass, and 8x faster, without needing extra modules or preprocessing.
Recent advances in 3D foundation models have led to growing interest in reconstructing humans and their surrounding environments. However, most existing approaches focus on monocular inputs, and extending them to multi-view settings requires additional overhead modules or preprocessed data. To this end, we present CHROMM, a unified framework that jointly estimates cameras, scene point clouds, and human meshes from multi-person multi-view videos without relying on external modules or preprocessing. We integrate strong geometric and human priors from Pi3X and Multi-HMR into a single trainable neural network architecture, and introduce a scale adjustment module to solve the scale discrepancy between humans and the scene. We also introduce a multi-view fusion strategy to aggregate per-view estimates into a single representation at test-time. Finally, we propose a geometry-based multi-person association method, which is more robust than appearance-based approaches. Experiments on EMDB, RICH, EgoHumans, and EgoExo4D show that CHROMM achieves competitive performance in global human motion and multi-view pose estimation while running over 8x faster than prior optimization-based multi-view approaches. Project page: https://nstar1125.github.io/chromm.
DeepMind