Search papers, labs, and topics across Lattice.
This paper formulates MDPs as an optimization problem over linear operators on general function spaces, enabling the application of perturbation theory to derive derivatives of the objective function. This operator-theoretic perspective generalizes reinforcement learning results previously limited to finite-state/action spaces or linear function approximations. The framework further yields novel, low-complexity PPO-type algorithms applicable to MDPs with general state and action spaces.
Generalizing RL to continuous state and action spaces just got easier: this paper introduces an operator-theoretic framework and PPO-type algorithms that ditch finite-state assumptions.
Markov decision processes (MDPs) is viewed as an optimization of an objective function over certain linear operators over general function spaces. Using the well-established perturbation theory of linear operators, this viewpoint allows one to identify derivatives of the objective function as a function of the linear operators. This leads to generalization of many well-known results in reinforcement learning to cases with generate state and action spaces. Prior results of this type were only established in the finite-state finite-action MDP settings and in settings with certain linear function approximations. The framework also leads to new low-complexity PPO-type reinforcement learning algorithms for general state and action space MDPs.
