Search papers, labs, and topics across Lattice.
This paper addresses the challenge of reinforcement learning in environments with adversarially corrupted feedback, specifically focusing on both reward and state corruption modeled by the Huber contamination model. The authors introduce the BR-Async-Q algorithm, which utilizes an epoch-based approach to batch online data, reducing variance and enabling robust estimates of the Bellman optimality operator. Their results show that BR-Async-Q achieves a high-probability error bound comparable to traditional Q-learning, marking a significant advancement in robustness guarantees for asynchronous Q-learning under corruption conditions.
Asynchronous Q-learning can now maintain performance even in the face of adversarially corrupted rewards and states, thanks to a novel batching approach.
Motivated by reinforcement learning in harsh environments, we consider the problem of learning an optimal policy subject to adversarially corrupted feedback. Specifically, at each time-step, an adversary can perturb both the reward and state observations of the learner following the Huber contamination model. To defend against such data corruption, we propose {\texttt{BR-Async-Q}}: a novel, epoch-based, robust \(Q\)-learning algorithm built upon two key ideas: (i) partitioning the online data stream into batches to reduce variance, and (ii) constructing robust estimates of the Bellman optimality operator using such batched data. We prove a high-probability $\ell_\infty$ error bound for {\texttt{BR-Async-Q}} that matches that for vanilla \(Q\)-learning, up to a small additive term that scales with the fraction of corrupted samples. To our knowledge, this provides the first robustness guarantee for asynchronous \(Q\)-learning subject to both reward and state corruption. Furthermore, when only rewards are corrupted, the dependence of our algorithm's bound on the corruption fraction is minimax optimal.
