Search papers, labs, and topics across Lattice.
This paper introduces Representation-Aware Advantage Estimation (GraphAE), which enhances reinforcement learning from human feedback (RLHF) by utilizing hidden states from reward models (RMs) as auxiliary signals for advantage estimation. By modeling sampled responses as a graph, where nodes represent responses and edges denote their similarity in the RM hidden space, GraphAE allows for improved contextual information propagation during advantage computation. Extensive experiments reveal that this method consistently improves performance across multiple benchmarks, achieving significant gains in sample efficiency and robustness in RLHF applications.
Leveraging hidden states from reward models can boost RLHF performance by over 6% on challenging benchmarks, transforming how we utilize reward signals.
Current reinforcement learning from human feedback (RLHF) methods primarily rely on scalar rewards from a trained reward model (RM). While effective, scalar rewards are often noisy and fail to capture fine-grained preference differences, whereas RM hidden states encode richer semantic and preference information. We introduce the representation-aware advantage estimation, which leverages RM hidden states and models them as auxiliary signals for better advantage estimation. Specifically, we propose the Graph-based Advantage Estimation (GraphAE), treat each sampled group as a graph, where nodes correspond to responses and edges capture their similarity in the RM hidden space. Then advantages are computed via graph propagation, enabling each sample to incorporate contextual information from its neighbors. GraphAE is lightweight and can be seamlessly integrated into existing group-based RL algorithms. We apply GraphAE to GRPO, GSPO and RLOO, and conduct extensive experiments on different models and benchmarks. Empirical results show consistent improvements across three benchmarks, with gains of up to + 6.3 on Arena-Hard-v0.1, + 8.27 on AlpacaEval 2.0, and + 0.22 on MT-Bench. These results demonstrate that leveraging RM representations leads to more sample efficient and robust RLHF.
