Search papers, labs, and topics across Lattice.
The paper introduces Preference Score Distillation (PSD), a novel optimization-based framework for aligning text-to-3D generation with human preferences using pre-trained 2D reward models, thereby circumventing the need for task-specific 3D training data. PSD reframes preference alignment as a classifier-free guidance (CFG)-style mechanism to address issues with pixel-level gradients from 2D reward models, drawing parallels with challenges in naive classifier guidance in conditioned diffusion models. The method also incorporates an adaptive co-optimization strategy for preference scores and negative text embeddings, enhancing alignment by dynamically refining negative text embeddings during optimization.
Forget task-specific fine-tuning: PSD aligns text-to-3D generation with human preferences using only pre-trained 2D reward models.
Human preference alignment presents a critical yet underexplored challenge for diffusion models in text-to-3D generation. Existing solutions typically require task-specific fine-tuning, posing significant hurdles in data-scarce 3D domains. To address this, we propose Preference Score Distillation (PSD), an optimization-based framework that leverages pretrained 2D reward models for human-aligned text-to-3D synthesis without 3D training data. Our key insight stems from the incompatibility of pixel-level gradients: due to the absence of noisy samples during reward model training, direct application of 2D reward gradients disturbs the denoising process. Noticing that similar issue occurs in the naive classifier guidance in conditioned diffusion models, we fundamentally rethink preference alignment as a classifier-free guidance (CFG)-style mechanism through our implicit reward model. Furthermore, recognizing that frozen pretrained diffusion models constrain performance, we introduce an adaptive strategy to co-optimize preference scores and negative text embeddings. By incorporating CFG during optimization, online refinement of negative text embeddings dynamically enhances alignment. To our knowledge, we are the first to bridge human preference alignment with CFG theory under score distillation framework. Experiments demonstrate the superiority of PSD in aesthetic metrics, seamless integration with diverse pipelines, and strong extensibility.
