Search papers, labs, and topics across Lattice.
This paper addresses the problem of average reward reinforcement learning in Semi-Markov Decision Processes (SMDPs) where rewards and durations are non-stationary. It introduces a novel modified harmonic mean operator to correctly compute reward rates in such environments, overcoming limitations of existing ratio-based approaches. The proposed method leads to model-free learning algorithms robust to non-stationary reward and duration distributions, and its efficacy is demonstrated both theoretically and empirically.
Average reward RL can finally handle the messy reality of non-stationary rewards and durations in SMDPs, thanks to a clever harmonic mean trick.
Recent research has revived and amplified interest in algorithms for undiscounted average reward reinforcement learning in infinite-horizon, non-episodic (continuing) tasks. Semi-Markov decision processes (SMDPs) are of particular interest. In SMDPs, discrete actions stochastically generate both rewards and durations, and the objective is to optimize the average reward rate. Existing algorithms approach this by optimizing the ratio of rewards to durations. However, when rewards and durations are non-stationary (in the infinite horizon), this can be incorrect. This paper presents a novel modified harmonic mean operator that correctly computes reward rates even under such conditions. This yields model-free learning algorithms that can work with SMDPs, while maintaining robustness to non-stationary reward and duration distributions over time. We prove theoretical properties of the modified harmonic mean operator, and empirically demonstrate its efficacy in comparison to existing algorithms.
