Search papers, labs, and topics across Lattice.
The paper introduces P-GenRM, a personalized generative reward model that addresses limitations in existing personalized reward models by transforming preference signals into structured evaluation chains to derive adaptive personas and scoring rubrics. P-GenRM clusters users into User Prototypes and employs a dual-granularity scaling mechanism, scaling at both the individual and prototype levels to mitigate noise and enhance generalization to unseen users. Empirical results demonstrate state-of-the-art performance on personalized reward model benchmarks, with a 2.31% average improvement, and a further 3% boost from test-time user-based scaling.
P-GenRM personalizes LLMs more effectively by generating adaptive personas and scoring rubrics from user preferences, outperforming existing reward models by 2.31% and offering a 3% boost via test-time scaling.
Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.
Tsinghua AI