Search papers, labs, and topics across Lattice.
The paper introduces MARS, a margin-aware reward modeling framework that adaptively augments training data by focusing on ambiguous preference pairs where the reward model exhibits low confidence. By concentrating augmentation on these low-margin samples, MARS iteratively refines the training distribution, effectively targeting the reward model's failure modes. Theoretical analysis demonstrates that MARS increases the average curvature of the loss function, while empirical results show consistent performance gains compared to uniform augmentation strategies.
Stop training your reward models on easy examples: MARS boosts reward modeling performance by focusing augmentation on the ambiguous preference pairs where the model struggles most.
Reward modeling is a core component of modern alignment pipelines including RLHF and RLAIF, underpinning policy optimization methods including PPO and TRPO. However, training reliable reward models relies heavily on human-labeled preference data, which is costly and limited, motivating the use of data augmentation. Existing augmentation approaches typically operate at the representation or semantic level and remain agnostic to the reward model's estimation difficulty. In this paper, we propose MARS, an adaptive, margin-aware augmentation and sampling strategy that explicitly targets ambiguous and failure modes of the reward model. Our proposed framework, MARS, concentrates augmentation on low-margin (ambiguous) preference pairs where the reward model is most uncertain, and iteratively refines the training distribution via hard-sample augmentation. We provide theoretical guarantees showing that this strategy increases the average curvature of the loss function hence enhance information and improves conditioning, along with empirical results demonstrating consistent gains over uniform augmentation for robust reward modeling.
