Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Endpoint Replay, a novel compression technique for experience replay in deep reinforcement learning that reduces the size of the recency buffer while maintaining performance. By storing only representative transitions from the end-points of connected n-step sequences, the method achieves a comparable memory horizon to traditional large buffers but requires significantly less storage. Empirical evaluations show that Endpoint Replay avoids the biases of naive compression methods and matches the performance of standard large buffers across multiple benchmarks, including the Pinball environment and Atari 2600.
Reducing the recency buffer size by an order of magnitude without sacrificing performance could revolutionize memory management in deep reinforcement learning.
Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected $n$-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.