Search papers, labs, and topics across Lattice.
This paper investigates a novel approach to value function estimation in reinforcement learning by dynamically learning the support bounds for a Gaussian-smoothed categorical target, rather than pre-defining them. The proposed method addresses the challenges posed by the non-stationary nature of target values in RL, allowing for a more flexible and adaptive learning process. Empirical results demonstrate that this approach not only matches but also improves upon existing histogram-based actor-critic algorithms in continuous-control tasks, while providing a tighter theoretical bound on the mean-squared Bellman error.
Dynamically learning support bounds for value functions can enhance stability and performance in reinforcement learning, outperforming traditional fixed-interval approaches.
Value functions are an essential component in actor-critic based deep reinforcement learning (RL). Conventionally, these functions are trained as a regression task by minimising the mean squared error (MSE) relative to bootstrapped target values. Meanwhile, in distributional RL, a distribution of returns is modelled based on the distributional Bellman operator. This work investigates the Gaussian Histogram Loss (HL-Gauss), a recent approach that reframes value estimation as classification by encoding each scalar Bellman target as a Gaussian-smoothed categorical target. Despite its potential, applying histogram-based losses to RL presents inherent challenges, most notably the requirement to pre-define a fixed support interval, which is often complicated by the non-stationary and stochastic nature of target values typically found in RL tasks. In this work, we propose an approach that dynamically learns the lower and upper bounds of the support instead of assigning them beforehand. We derive an objective that jointly learns these bounds whilst learning the categorical representation of the scalar values, and we show that this objective forms an upper bound on the mean-squared Bellman error. Our theoretical analysis further shows that this bound is tighter than that of non-learned supports of HL-Gauss. Empirically, the proposed objective enables stable adaptation of the support interval and matches HL-Gauss-based actor-critic algorithms on most continuous-control tasks whilst improving on a subset, without requiring a pre-specified support interval.
