Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces BP-LLM, a probabilistic framework that enhances binary feedback alignment in large language models by employing belief propagation to model uncertainty in feedback. By treating binary comparisons as noisy observations of latent reward margins, BP-LLM refines beliefs through stable message exchanges between a classifier and the policy, outperforming existing methods like Direct Preference Optimization (DPO) and Binary Classifier Optimization (BCO) in both inference and training settings. The results demonstrate significant improvements in win rates, particularly under noisy or heterogeneous feedback conditions, highlighting the framework's robustness and scalability.
BP-LLM achieves superior alignment performance by effectively modeling uncertainty in binary feedback, outperforming traditional methods even in noisy environments.
Preference-based alignment methods such as Direct Preference Optimization (DPO) and Binary Classifier Optimization (BCO) offer efficient alternatives to reinforcement learning from human feedback (RLHF). However, they often treat comparisons as independent labels and do not explicitly model uncertainty, which can lead to miscalibration and reduced robustness under noisy or heterogeneous feedback. We introduce Belief Propagation for Large Language Model Alignment (BP–LLM), a probabilistic framework that views binary feedback as noisy observations of latent reward margins under a policy-induced Gaussian prior. Using the Jaakkola–Jordan variational bound, BP–LLM yields closed-form Gaussian message updates and performs stable belief propagation between a classifier-side inference module and the policy. Exchanging extrinsic messages enables both modules to refine beliefs without double counting and recovers BCO and DPO as special cases. We evaluate BP–LLM in two regimes. In an inference-only setting with frozen LLM weights, BP–LLM consistently improves label-free test-time win rate over BCO and DPO across UltraFeedback, Capybara, and HelpSteer2 for open-weight Llama and Qwen models. In a training-time setting with parameter-efficient LoRA updates, BP–LLM also outperforms Cal-DPO with higher win rates. Overall, BP–LLM is most beneficial under noisy/heterogeneous (or unary) feedback, where posterior refinement and extrinsic shaping provide more reliable signals than hard labels, while remaining lightweight and scalable.1