Search papers, labs, and topics across Lattice.
This paper introduces the VLM-PBRS framework, which automates potential-based reward shaping (PBRS) by leveraging feedback from vision language models (VLMs) to learn potential functions without the need for expert-designed heuristics. The approach addresses the challenges of sparse rewards in reinforcement learning by using preferences from a lightweight VLM to inform the potential function, thus preserving optimal policies while enhancing sample efficiency. Empirical results in the Meta-World and Franka Kitchen environments demonstrate that even with less accurate preference labels from smaller VLMs, the method significantly accelerates learning and mitigates reward hacking issues.
Automating potential-based reward shaping with VLM guidance not only preserves optimal policies but also boosts sample efficiency, even with less accurate feedback.
Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision language model (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.
