Search papers, labs, and topics across Lattice.
The paper addresses the issue of unnatural poses in single-view 3D human reconstruction from multi-view diffusion models, particularly for dynamic poses, due to limited training data. They introduce DrPose, a direct reward fine-tuning algorithm that optimizes a multi-view diffusion model using a PoseScore reward function, which measures consistency between generated multi-view latent images and ground-truth human poses. By training on the newly constructed DrPose15K dataset, which contains diverse poses, DrPose achieves improved pose accuracy and overall reconstruction quality on both standard benchmarks and challenging in-the-wild images.
Forget expensive 3D human datasets: DrPose directly fine-tunes multi-view diffusion models using only single-view images and human poses, leading to more natural 3D human reconstructions, particularly for dynamic poses.
Single-view 3D human reconstruction has achieved remarkable progress through the adoption of multi-view diffusion models, yet the recovered 3D humans often exhibit unnatural poses. This phenomenon becomes pronounced when reconstructing 3D humans with dynamic or challenging poses, which we attribute to the limited scale of available 3D human datasets with diverse poses. To address this limitation, we introduce DrPose, Direct Reward fine-tuning algorithm on Poses, which enables post-training of a multi-view diffusion model on diverse poses without requiring expensive 3D human assets. DrPose trains a model using only human poses paired with single-view images, employing a direct reward fine-tuning to maximize PoseScore, which is our proposed differentiable reward that quantifies consistency between a generated multi-view latent image and a ground-truth human pose. This optimization is conducted on DrPose15K, a novel dataset that was constructed from an existing human motion dataset and a pose-conditioned video generative model. Constructed from abundant human pose sequence data, DrPose15K exhibits a broader pose distribution compared to existing 3D human datasets. We validate our approach through evaluation on conventional benchmark datasets, in-the-wild images, and a newly constructed benchmark, with a particular focus on assessing performance on challenging human poses. Our results demonstrate consistent qualitative and quantitative improvements across all benchmarks. Project page: https://seunguk-do.github.io/drpose.
