Search papers, labs, and topics across Lattice.
This paper establishes the first finite sample complexity guarantees for learning policies in weakly communicating average-reward Markov Decision Processes (MDPs) using only a single trajectory. By analyzing the dynamics of such trajectories, the authors develop novel model-free methods that achieve finite sample complexity of $\widetilde{O}(1/\varepsilon^2)$ for value-based methods and $\widetilde{O}(1/\varepsilon^4)$ for policy-based methods. Importantly, their approach does not require prior knowledge of problem-dependent quantities, marking a significant advancement in the field.
Learning policies from just one trajectory in average-reward MDPs is now feasible, with guarantees that could transform how we approach sample efficiency in reinforcement learning.
While there is an extensive body of work characterizing the sample complexity of discounted cumulative-reward MDPs, finite sample analyses for average-reward MDPs have been limited, and most existing works rely on restrictive assumptions such as ergodicity or access to a generative model. In this work, we establish the first finite sample complexity guarantees from a single trajectory for weakly communicating average-reward MDPs. To this end, we study the dynamics of a single trajectory in weakly communicating MDPs and based on this analysis, we develop novel model-free methods. Notably, our value-based and policy-based methods provide finite sample complexity guarantees of $\widetilde{O}(1/\varepsilon^2)$ and $\widetilde{O}(1/\varepsilon^4)$ from a single trajectory in weakly communicating MDPs, respectively. Furthermore, we introduce the first model-free method that requires no prior knowledge of problem-dependent quantities for communicating MDPs.
