Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a generalized temporal-difference reinforcement learning framework that leverages conditional expectations to treat value and action-value functions as uncertain quantities. By moving beyond the linear-Gaussian assumptions of classical Kalman-based methods, the proposed approach accommodates nonlinear models and non-Gaussian distributions, allowing for a more robust estimation of value functions and their uncertainty. The framework is validated through numerical examples on optimal control problems, demonstrating its effectiveness in accurately estimating both the value function and its associated uncertainty in complex stochastic environments.
Uncertainty in value function estimation can be quantified and leveraged for more robust reinforcement learning across a wider range of stochastic systems.
In this paper, we present a generalized temporal-difference (TD) reinforcement learning framework based on the theory of conditional expectations. The value and action-value (Q-value) functions are treated as uncertain quantities, and their estimation is formulated as a stochastic inference problem. Unlike classical Kalman-based temporal-difference learning, which relies on linear-Gaussian assumptions, the proposed formulation is derived directly from the conditional expectation framework and naturally extends to nonlinear models and non-Gaussian probability distributions. The proposed method recursively estimates not only the conditional expectation of the value function but also its second probabilistic moment, thereby quantifying the uncertainty associated with the learned value function throughout the learning process. To obtain a computationally tractable algorithm, the stochastic problem is discretized using either polynomial chaos expansions or ensemble-based approximations, providing efficient representations of the underlying random variables. The proposed framework is demonstrated on two optimal control problems: a linear mass--spring--damper system and a nonlinear heat conduction problem in a closed cavity. The numerical examples illustrate the capability of the proposed method to accurately estimate both the value function and its associated uncertainty, while extending classical Kalman-based temporal-difference learning to a broader class of stochastic systems.