On Symmetric Losses for Policy Optimization with Noisy Preferences

It is proved that symmetric losses enable successful policy improvement even with noisy labels, as the resulting reward is rank-preserving—a property that is identified as sufficient for policy improvement.