Search papers, labs, and topics across Lattice.
This paper introduces a difficulty-based data selection strategy for preference datasets used in aligning LLMs, leveraging the DPO implicit reward gap. By prioritizing preference pairs with smaller reward gaps, the method focuses training on more challenging examples. Experiments across multiple datasets and alignment tasks demonstrate that training on only 10% of the data selected by this method outperforms training on the full dataset and several strong baselines.
Train better aligned LLMs with 10% of the data by strategically focusing on the most difficult preference comparisons.
Aligning large language models (LLMs) with human preferences is a critical challenge in AI research. While methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are widely used, they often rely on large, costly preference datasets. The current work lacks methods for high-quality data selection specifically for preference data. In this work, we introduce a novel difficulty-based data selection strategy for preference datasets, grounded in the DPO implicit reward mechanism. By selecting preference data examples with smaller DPO implicit reward gaps, which are indicative of more challenging cases, we improve data efficiency and model alignment. Our approach consistently outperforms five strong baselines across multiple datasets and alignment tasks, achieving superior performance with only 10\% of the original data. This principled, efficient selection method offers a promising solution for scaling LLM alignment with limited resources.
UW