Search papers, labs, and topics across Lattice.
This paper introduces Decision-Directed Optimization for Reward Modeling (DDO-RM), an alternative to Direct Preference Optimization (DPO) for aligning LLMs with human preferences. DDO-RM reframes preference optimization as a decision problem, creating a policy distribution over candidate responses and distilling a reward-guided target distribution back into the policy. Experiments on EleutherAI/pythia-410m using HuggingFaceH4/ultrafeedback\_binarized show that DDO-RM achieves higher mean pair accuracy, AUC, and mean margin compared to DPO on a held-out test split.
DPO might not be the only game in town: a decision-directed approach to reward modeling can outperform it in pairwise preference optimization.
This paper reorganizes the current manuscript around the DPO versus DDO-RM preference-optimization project and focuses on two parts: the algorithmic view and the preliminary held-out benchmark. The benchmark asks a narrow question: even in a minimal pairwise chosen-versus-rejected setting, can a reward-guided decision-distribution update outperform a direct pairwise objective? We compare Direct Preference Optimization (DPO) against DDO-RM on EleutherAI/pythia-410m using HuggingFaceH4/ultrafeedback\_binarized, evaluate on the held-out test\_prefs split, and report results for seeds 42, 13, and 3407. Algorithmically, DDO-RM treats each prompt as a finite decision problem over candidate responses. Instead of optimizing only a binary chosen-rejected relation, it forms a policy distribution over candidates, centers reward-model scores under that distribution, and distills a reward-guided target distribution back into the policy. In the current public benchmark, DDO-RM improves mean pair accuracy from 0.5238 to 0.5602, AUC from 0.5315 to 0.5382, and mean margin from 0.1377 to 0.5353 relative to DPO. These are encouraging but still preliminary results: the study covers one model family, one dataset, one held-out evaluation split, and three seeds.
CMU ML
Tsinghua AI