Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces "intentional updates" for streaming reinforcement learning, a novel approach that directly controls the function output change during updates to improve stability. They propose Intentional TD, which aims for a fixed fractional reduction of the TD error, and Intentional Policy Gradient, which bounds the per-step policy change using KL divergence. Experiments show that these methods achieve state-of-the-art streaming performance, matching the performance of batch and replay-buffer methods.
Streaming RL can finally compete with batch RL, thanks to a new update rule that directly controls function output change and avoids instability.
In gradient-based learning, a step size chosen in parameter units does not produce a predictable per-step change in function output. This often leads to instability in the streaming setting (i.e., batch size=1), where stochasticity is not averaged out and update magnitudes can momentarily become arbitrarily big or small. Instead, we propose intentional updates: first specify the intended outcome of an update and then solve for the step size that approximately achieves it. This strategy has precedent in online supervised linear regression via Normalized Least Mean Squares algorithm, which selects a step size to yield a specified change in the function output proportional to the current error. We extend this principle to streaming deep reinforcement learning by defining appropriate intended outcomes: Intentional TD aims for a fixed fractional reduction of the TD error, and Intentional Policy Gradient aims for a bounded per-step change in the policy, limiting local KL divergence. We propose practical algorithms combining eligibility traces and diagonal scaling. Empirically, these methods yield state-of-the-art streaming performance, frequently performing on par with batch and replay-buffer approaches.