Search papers, labs, and topics across Lattice.
The paper introduces Variational Reward Modeling (VRM), a new framework for training reward models that explicitly models the human evaluation process by incorporating latent variables representing high-dimensional objective weights and low-dimensional semantic features. VRM uses variational inference to infer these latent variables, leading to a reward model that is less susceptible to capturing spurious correlations and more aligned with authentic human preferences. Experiments on benchmark datasets demonstrate that VRM outperforms existing methods in capturing human preferences, and a theoretical analysis provides a tighter generalization error bound.
By explicitly modeling the latent human evaluation process, VRM offers a more robust reward model, sidestepping the pitfalls of spurious correlations that plague traditional methods.
Large Language Models (LLMs) have achieved remarkable success across diverse natural language tasks, yet the reward models employed for aligning LLMs often encounter challenges of reward hacking, where the approaches predominantly rely on directly mapping prompt-response pairs to scalar scores, which may inadvertently capture spurious correlations rather than authentic human preferences. In contrast, human evaluation employs a sophisticated process that initially weighs the relative importance of multiple high-dimensional objectives according to the prompt context, subsequently evaluating response quality through low-dimensional semantic features such as logical coherence and contextual appropriateness. Motivated by this consideration, we propose VRM, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques. Additionally, we provide a theoretical analysis showing that VRM can achieve a tighter generalization error bound compared to the traditional reward model. Extensive experiments on benchmark datasets demonstrate that VRM significantly outperforms existing methods in capturing authentic human preferences.
