Search papers, labs, and topics across Lattice.
The paper introduces Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC), a novel inference-time method to address the brittleness of preference-based alignment methods caused by heterogeneous human preferences. DARC reranks candidate responses by maximizing a KL-robust satisfaction objective, effectively capping or penalizing entropic risk relative to the mean preference. Experiments on alignment benchmarks demonstrate that DARC reduces disagreement and tail risk, while maintaining competitive average quality, without requiring retraining.
Mitigate the brittleness of RLHF by explicitly controlling for disagreement and tail risk during inference, without retraining, using a KL-robust optimization framework.
Preference-based alignment methods (e.g., RLHF, DPO) typically optimize a single scalar objective, implicitly averaging over heterogeneous human preferences. In practice, systematic annotator and user-group disagreement makes mean-reward maximization brittle and susceptible to proxy over-optimization. We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**, a retraining-free inference-time method that frames response selection as distributionally robust, risk-sensitive decision making. Given multiple preference samples or scalable disagreement proxies, DARC reranks candidates by maximizing a *KL-robust (entropic)* satisfaction objective, and provides simple deployment controls that cap or penalize the corresponding entropic risk premium relative to the mean, enabling explicit risk budgets without retraining. We provide theoretical characterization linking this decoding rule to principled pessimism and KL-based distributionally robust optimization. Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality under noisy, heterogeneous feedback.
