Search papers, labs, and topics across Lattice.
LatRef-Diff, a novel diffusion-based framework, is introduced for precise facial attribute editing and style manipulation by replacing traditional semantic directions with style codes generated via latent and reference guidance. A style modulation module, incorporating learnable vectors, cross-attention, and a hierarchical design, integrates these style codes into the target image. To enhance training stability and eliminate the need for paired images, a forward-backward consistency training strategy is proposed, which removes and then restores the target attribute using image-specific semantic directions and style modulation.
Achieve state-of-the-art facial attribute editing and style manipulation with a diffusion model by ditching semantic directions for style codes and a clever forward-backward consistency training strategy that avoids paired images.
Facial attribute editing and style manipulation are crucial for applications like virtual avatars and photo editing. However, achieving precise control over facial attributes without altering unrelated features is challenging due to the complexity of facial structures and the strong correlations between attributes. While conditional GANs have shown progress, they are limited by accuracy issues and training instability. Diffusion models, though promising, face challenges in style manipulation due to the limited expressiveness of semantic directions. In this paper, we propose LatRef-Diff, a novel diffusion-based framework that addresses these limitations. We replace the traditional semantic directions in diffusion models with style codes and propose two methods for generating them: latent and reference guidance. Based on these style codes, we design a style modulation module that integrates them into the target image, enabling both random and customized style manipulation. This module incorporates learnable vectors, cross-attention mechanisms, and a hierarchical design to improve accuracy and image quality. Additionally, to enhance training stability while eliminating the need for paired images (e.g., before and after editing), we propose a forward-backward consistency training strategy. This strategy first removes the target attribute approximately using image-specific semantic directions and then restores it via style modulation, guided by perceptual and classification losses. Extensive experiments on CelebA-HQ demonstrate that LatRef-Diff achieves state-of-the-art performance in both qualitative and quantitative evaluations. Ablation studies validate the effectiveness of our model's design choices.
