Search papers, labs, and topics across Lattice.
This paper investigates the role of pessimism in offline reinforcement learning (RL) and its impact on generalization within contextual MDPs (CMDPs). The authors demonstrate that the structure of pessimism, rather than its quantity, is crucial for achieving optimal generalization, revealing that a symmetric value function can outperform a non-symmetric one even when the latter is more pessimistic. Empirical validation through IQL and CQL in a rotationally symmetric reacher environment supports the claim that applying data augmentation via a consistency loss during policy extraction enhances generalization performance.
Overly pessimistic approaches can still achieve optimal generalization in offline RL, but only if they respect the underlying symmetries of the solution space.
While pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization. However, in this paper we demonstrate that being overly pessimistic does not inherently prevent optimal generalization in contextual MDPs (CMDPs). Instead, we argue successful generalization depends not on the amount of pessimism, but whether the pessimistic structure respects the underlying symmetries of the optimal solution. We prove that a mildly pessimistic, non-symmetric value function can generalize worse than an overly pessimistic, symmetric one. In offline RL, the structure of the pessimism is determined by the structure of the dataset coverage. As such, enforcing a symmetric value function can be non-trivial, and might require techniques such as data augmentation (DA). Inspired by our theoretical results, we argue that DA can best be applied through a consistency loss during policy extraction, rather than the common practice of (regular) offline training on an augmented dataset. This is empirically validated using IQL and CQL on a rotationally symmetric reacher environment.
