Search papers, labs, and topics across Lattice.
The paper introduces Visited-State Augmentation (ViSA), a novel data augmentation technique for Contrastive Reinforcement Learning (CRL) that improves goal-space generalization. ViSA generates augmented state samples to address the limitation of CRL in accurately estimating the value function for hard-to-visit goals, and regularizes the embedding space by reformulating the objective function based on mutual information. Experiments in simulation and real-world robotic tasks demonstrate that ViSA enhances goal-space generalization, leading to more accurate value estimation for challenging goals.
CRL struggles with hard-to-reach goals, but ViSA, a new data augmentation technique, solves this by generating synthetic states and regularizing the embedding space, leading to better value estimation.
Goal-Conditioned Reinforcement Learning (GCRL) is a framework for learning a policy that can reach arbitrarily given goals. In particular, Contrastive Reinforcement Learning (CRL) provides a framework for policy updates using an approximation of the value function estimated via contrastive learning, achieving higher sample efficiency compared to conventional methods. However, since CRL treats the visited state as a pseudo-goal during learning, it can accurately estimate the value function only for limited goals. To address this issue, we propose a novel data augmentation approach for CRL called ViSA (Visited-State Augmentation). ViSA consists of two components: 1) generating augmented state samples, with the aim of augmenting hard-to-visit state samples during on-policy exploration, and 2) learning consistent embedding space, which uses an augmented state as auxiliary information to regularize the embedding space by reformulating the objective function of the embedding space based on mutual information. We evaluate ViSA in simulation and real-world robotic tasks and show improved goal-space generalization, which permits accurate value estimation for hard-to-visit goals. Further details can be found on the project page: \href{https://issa-n.github.io/projectPage_ViSA/}{\texttt{https://issa-n.github.io/projectPage\_ViSA/}}
