Search papers, labs, and topics across Lattice.
This paper identifies and addresses procedural fairness failures in Reinforcement Learning from Human Feedback (RLHF) due to the assumption of preference homogeneity, which leads to majority preferences dominating the reward model. By introducing Preference-Aware RLHF (PA-RLHF), the authors separate optimization across distinct preference modes, resulting in a significant improvement in alignment accuracy from 46.9% to 67.9% and a reduction in the fairness gap among preference groups. These findings highlight the critical impact of reward modeling design on fairness in AI systems, particularly in applications involving large language models and agentic systems.
Majority preferences can skew reward models in RLHF, but a new approach boosts alignment accuracy and fairness for minority voices.
Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity. When preferences are heterogeneous, this aggregation induces a procedural fairness failure where majority preference groups dominate reward learning while minority preferences are systematically under-represented. This work defines procedural fairness in alignment as preserving distinct preference signals during reward modeling and shows that standard RLHF violates this via preference averaging. Preference-Aware RLHF (PA-RLHF) is introduced, separating optimization across preference modes at the reward learning stage. In a controlled setting, PA-RLHF improves overall alignment accuracy from 46.9% to 67.9% and reduces the fairness gap between best and worst aligned groups from 15.9 to 9.6 percentage points. These results show that procedural fairness failures in alignment can arise from structural design choices in reward learning, even in controlled, noise-free settings, with direct implications for large language models and agentic systems, where biased reward models can compound inequities across sequential decisions.
