Search papers, labs, and topics across Lattice.
This paper introduces a conditional denoising diffusion model for one-shot human image synthesis that leverages 3D human priors to address limitations in existing pose transfer and generalizable NeRF methods. By incorporating 3D normal maps and color prompts, the model effectively synthesizes complex human poses while accurately reconstructing occluded or invisible parts. Experimental results show that this approach significantly outperforms prior methods and demonstrates improved generalization across various datasets.
Achieving high-quality human image synthesis from a single reference image, this method excels in generating complex poses and recovering occluded details.
This paper addresses the challenge of one-shot novel view and pose human image synthesis. The existing methods transfer the reference human image to a target pose using a set of 2D pose keypoints or synthesize human images based on generalizable human NeRF which uses human model priors to extract point-wise features. However, pose transfer based methods can not handle complex human pose using ambiguous 2D pose as the condition, while generalizable human NeRFs may be inaccurate to recover occluded/invisiable human parts without extracted reliable features. To solve these problems, we propose a novel approach for novel view and pose synthesis from a singe human image via conditional denoising diffusion model. Our diffusion model divides the novel view and pose synthesis problem into a sequence of conditional denoising steps. Specifically, to generate humans with complex and arbitrary poses, we introduce 3D human priors, i.e., 3D normal map and color prompt, as geometry and color conditions into the generation process. By transferring the reference human into the target human with a series of diffusion steps, our diffusion model enables high-quality synthesis including the occluded/invisible parts. Further, we propose a self-reconstruction based customized refinement to enhance fine details when tested on novel persons.Experimental results on different public datasets demonstrate that our approach significantly outperforms previous methods and also shows better generalization ability across datasets. The code will be made publicly available at https://github.com/Yankeegsj/3DPGDM.
