Search papers, labs, and topics across Lattice.
The paper introduces SEAR, a novel off-policy online reinforcement learning algorithm designed to improve sample efficiency in action chunking. SEAR addresses the challenges of critic approximation and data efficiency in long-horizon tasks by exploiting the temporal structure of action chunks and employing a receding horizon approach. The algorithm demonstrates superior performance compared to state-of-the-art online RL methods on Metaworld, even with chunk sizes up to 20.
Action chunking, previously limited to offline RL, can now achieve state-of-the-art performance in online settings thanks to SEAR's receding horizon approach.
Action chunking can improve exploration and value estimation in long horizon reinforcement learning, but makes learning substantially harder since the critic must evaluate action sequences rather than single actions, greatly increasing approximation and data efficiency challenges. As a result, existing action chunking methods, primarily designed for the offline and offline-to-online settings, have not achieved strong performance in purely online reinforcement learning. We introduce SEAR, an off policy online reinforcement learning algorithm for action chunking. It exploits the temporal structure of action chunks and operates with a receding horizon, effectively combining the benefits of small and large chunk sizes. SEAR outperforms state of the art online reinforcement learning methods on Metaworld, training with chunk sizes up to 20.
