Search papers, labs, and topics across Lattice.
This paper addresses the challenge of policy optimization in online episodic tabular Markov decision processes (MDPs) with unknown transition kernels by developing a novel algorithm that leverages optimistic follow-the-regularized-leader. The authors achieve data-dependent regret bounds that adapt to both adversarial and stochastic losses, filling a critical gap in existing literature which only considered known transitions. Key findings include the introduction of new optimistic $Q$-function estimators and a data-dependent transition bonus that effectively manage estimator bias, leading to improved regret performance characterized by first-order, second-order, and path-length bounds alongside a transition-dependent complexity term.
Achieving data-dependent regret bounds in policy optimization with unknown transitions could redefine our understanding of adaptive learning in MDPs.
We study policy optimization for online episodic tabular Markov decision processes with unknown transition kernels, aiming for best-of-both-worlds guarantees together with data-dependent regret bounds. Recent work (Dann et al., 2023; Li et al., 2026) has shown that policy optimization can adapt to both adversarial and stochastic losses with first-order, second-order, and path-length bounds, but only under known transitions, leaving open whether such data-dependent guarantees are achievable by policy optimization when the transition kernel is unknown. We resolve this by developing a new algorithm based on optimistic follow-the-regularized-leader that attains these guarantees under unknown transitions. The key ingredient is a new design of optimistic $Q$-function estimators together with a data-dependent transition bonus that controls estimator bias through the loss-prediction error. Our analysis further identifies an unavoidable transition-dependent complexity term that captures the intrinsic cost of estimating the transition kernel. As a result, we obtain first-order, second-order, and path-length bounds with the transition-dependent complexity term while simultaneously achieving gap-dependent $\mathrm{polylog}(T)$ regret in the stochastic regime.
