Search papers, labs, and topics across Lattice.
This paper introduces Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO), a novel approach to tackle the performance degradation in reinforcement learning caused by delayed feedback in stochastic environments. By theoretically establishing the discrepancy between delayed and true states, the authors demonstrate how this gap negatively impacts optimal policy performance. DUPO utilizes a diffusion model to explicitly model this relationship and effectively weights delayed policies, leading to superior performance in continuous robotic control tasks compared to existing methods, even under significant and random delays.
Delayed feedback in reinforcement learning can be effectively managed by modeling discrepancies with diffusion techniques, leading to improved policy performance in challenging environments.
Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.
