Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We address episodic adversarial Markov Decision Processes (MDPs) with stochastic hard constraints, aiming for optimal regret from a known strictly feasible policy with margin $d$. Our proposed algorithm, MA-OPS, achieves a regret of $\widetilde{\mathcal{O}}(\sqrt{T}/\rho + 1/(d\rho))$, which is optimal with respect to $T$, $d$, and $\rho$. This is accomplished by combining optimistic search for the Slater margin with pessimistic policy evaluation to learn a policy with a large feasibility margin that satisfies constraints episode-wise.
Optimal regret in adversarial MDPs with stochastic hard constraints is finally achieved, bridging the gap between existing upper and lower bounds by achieving $\widetilde{\mathcal{O}}(\sqrt{T}/\rho + 1/(d\rho))$ regret.
We study episodic constrained Markov decision processes with adversarial losses under stochastic hard constraints. Specifically, starting from a known strictly feasible policy with margin $d$, we seek to obtain optimal regret while satisfying the expected cost constraints in every episode. In this setting, Stradi et al. (2025) show that a carefully designed mixing rule attains regret of order $\widetilde{\mathcal{O}}(\sqrt{T}/\min\{d,d^2\})$. Interestingly, they also provide a lower bound of order $\Omega(\sqrt{T}/\rho)$ for the same setting, where $\rho$ is the Slater margin of the offline problem and can be much larger than $d$. In this work, we build on their approach to obtain optimal regret dependence on these margins. Specifically, we propose MA-OPS, an algorithm that combines an optimistic search for the Slater margin with a pessimistic evaluation of the selected policies to safely learn a policy with a large feasibility margin. This policy is then used to minimize regret while satisfying the constraints at every episode. In particular, we show that MA-OPS attains regret $\widetilde{\mathcal{O}}(\sqrt{T}/\rho + 1/(d\rho))$. Finally, we provide a matching lower bound, showing that the dependence on $T$, $d$, $\rho$ in the regret bound is optimal up to logarithmic factors.