Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Relative Value Learning (RV), a novel framework in reinforcement learning that focuses on learning value differences rather than absolute state values. By employing an antisymmetric function to directly estimate these differences, the authors derive a pairwise Bellman operator that guarantees convergence to the true value differences and reconstruct a generalized advantage estimation method. The integration of RV with Proximal Policy Optimization (PPO) demonstrates competitive performance on the Atari benchmark, suggesting that relative value estimation can effectively replace traditional absolute critics.
Learning value differences, rather than absolute values, can lead to more effective reinforcement learning strategies, as shown by RV's competitive performance on Atari games.
In reinforcement learning, critics typically estimate absolute state values $V(s)$, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function $Δ(s_i, s_j) = V(s_i) - V(s_j)$. We introduce a pairwise Bellman operator and prove it is a $γ$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step, $n$-step and $λ$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.