Search papers, labs, and topics across Lattice.
This paper introduces PAPA, a novel method for optimizing diffusion models in personalized recommender systems by directly leveraging real-time user feedback, thus eliminating the need for a parameterized reward model. The approach frames the problem as a reinforcement learning task, enabling efficient preference alignment without the extensive preference data typically required. Extensive experiments demonstrate that PAPA, along with its enhanced fine-tuning strategy EPAPA, significantly improves performance across various alignment tasks while reducing computational costs.
Real-time user feedback can now directly optimize diffusion models for personalized recommendations, eliminating the need for extensive preference data.
Diffusion models are highly effective at modeling complex data distributions, including images and text. However, in applications like personalized recommender systems, the objective often shifts to modeling specific regions of the distribution that maximize user preferences-initially unknown but gradually uncovered through interactive feedback. This can naturally be framed as a reinforcement learning problem, where the goal is to fine-tune a diffusion model to maximize a reward function based on preferences. However, the main challenge lies in learning a parameterized reward model, which typically requires large-scale preference data-something that is often not feasible in practice. In this work, we introduce Personalized Active Preference Alignment PAPA, a novel method that bypasses the requirement for a parametrized reward model by directly optimizing the diffusion model using real-time user feedback. PAPA enables feedback-efficient preference alignment, drawing inspiration from the variational inference framework. We demonstrate PAPA's effectiveness through extensive experiments and ablation studies across diverse class-conditioned and fine-grained alignment tasks. Additionally, based on theoretical insights, we propose an enhanced fine-tuning strategy, referred to as EPAPA, that requires less computational budget and accelerates the fine-tuning process, further boosting PAPA's suitability for real-world deployment. Our code is made publicly available at https://github.com/NasikNafi/papa.
