Search papers, labs, and topics across Lattice.
This paper explores analytic planning in model-based reinforcement learning under uncertainty by leveraging a quadratic action-value parameterization to simplify the Bellman backup process. By establishing a compatibility principle between predictive transition distributions and value function classes, the authors derive a closed-form backup that effectively propagates both mean and covariance of state distributions. The empirical results demonstrate significant reductions in target variance and improved calibration of predictive uncertainty in continuous control tasks, highlighting the potential of distribution-aware planning without restrictive assumptions.
Target variance in reinforcement learning can be drastically reduced by analytically propagating uncertainty without restrictive policy structures.
Effective model-based reinforcement learning in stochastic environments requires planning that accounts for predictive uncertainty. Propagating full state distributions analytically offers a principled way to do this, but has traditionally required restrictive policy or reward structures to remain tractable. Consequently, modern deep reinforcement learning has largely retreated to either stochastic sampling, which introduces significant target variance, or deterministic point estimates that ignore predictive covariance entirely. We investigate whether distribution-aware planning is possible without these constraints. Using a quadratic action-value parameterization, we first reduce the Bellman backup to an expectation over the state-value function alone; the key idea is then a compatibility principle between the predictive transition distribution and the value function class, under which this expectation is analytic in the distribution's moments. We instantiate this principle with a Gaussian transition model paired with a radial-basis value function, yielding a closed-form backup that propagates both predictive mean and covariance. Empirically, our approach reduces target variance and yields well-calibrated predictive uncertainty under stochastic observations in continuous control, providing a principled framework for planning with learned distribution models.
Mila