Search papers, labs, and topics across Lattice.
This paper extends contrastive reinforcement learning (CRL) by modeling action sequences as chunks rather than relying solely on single-step actions. The authors demonstrate that this approach leads to substantial performance improvements, achieving gains of +31.7% and +93.1% across 18 and 11 environments, respectively. The findings indicate that action chunks provide richer information about goals, enhancing the critic's representations and overall algorithm effectiveness.
Action chunking in CRL boosts performance by over 90% in certain environments, revealing that richer temporal representations can significantly enhance learning.
While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.
