Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel actor-critic algorithm, Pseudo-Quantized Actor-Critic (PQAC), designed to be robust against noisy temporal difference (TD) errors by revisiting TD learning through the lens of control as inference. PQAC models the distribution of optimality using a sigmoid function and leverages forward and reverse Kullback-Leibler divergences to derive a learning rule that implicitly excludes large, potentially noisy TD errors from the learning process. By decomposing optimality into multiple levels for pseudo-quantization of TD errors and approximating a Jensen-Shannon divergence-based approach, PQAC achieves more stable learning, especially in environments with noisy rewards or when traditional heuristics fail.
Achieve stable RL in noisy environments by "pseudo-quantizing" temporal difference errors, effectively ignoring large, potentially spurious updates.
In reinforcement learning (RL), temporal difference (TD) errors are widely adopted for optimizing value and policy functions. However, since the TD error is defined by a bootstrap method, its computation tends to be noisy and destabilize learning. Heuristics to improve the accuracy of TD errors, such as target networks and ensemble models, have been introduced so far. While these are essential approaches for the current deep RL algorithms, they cause side effects like increased computational cost and reduced learning efficiency. Therefore, this paper revisits the TD learning algorithm based on control as inference, deriving a novel algorithm capable of robust learning against noisy TD errors. First, the distribution model of optimality, a binary random variable, is represented by a sigmoid function. Alongside forward and reverse Kullback-Leibler divergences, this new model derives a robust learning rule: when the sigmoid function saturates with a large TD error probably due to noise, the gradient vanishes, implicitly excluding it from learning. Furthermore, the two divergences exhibit distinct gradient-vanishing characteristics. Building on these analyses, the optimality is decomposed into multiple levels to achieve pseudo-quantization of TD errors, aiming for further noise reduction. Additionally, a Jensen-Shannon divergence-based approach is approximately derived to inherit the characteristics of both divergences. These benefits are verified through RL benchmarks, demonstrating stable learning even when heuristics are insufficient or rewards contain noise.