Search papers, labs, and topics across Lattice.
This paper introduces Multi-View Video Reward Shaping (MVR), a novel reinforcement learning framework that leverages video-text similarity from a frozen pre-trained VLM to learn a state relevance function for reward shaping. MVR addresses limitations of image-based reward shaping by using multi-view videos to capture complex, dynamic motions and mitigate biases towards specific static poses. The framework also incorporates a state-dependent reward shaping formulation that dynamically adjusts the influence of VLM guidance, leading to improved performance on humanoid locomotion and manipulation tasks.
Forget hand-crafted reward functions: MVR uses multi-view video and a frozen VLM to automatically shape RL rewards, teaching agents complex motions without getting stuck on static poses.
Reward design is of great importance for solving complex tasks with reinforcement learning. Recent studies have explored using image-text similarity produced by vision-language models (VLMs) to augment rewards of a task with visual feedback. A common practice linearly adds VLM scores to task or success rewards without explicit shaping, potentially altering the optimal policy. Moreover, such approaches, often relying on single static images, struggle with tasks whose desired behavior involves complex, dynamic motions spanning multiple visually different states. Furthermore, single viewpoints can occlude critical aspects of an agent's behavior. To address these issues, this paper presents Multi-View Video Reward Shaping (MVR), a framework that models the relevance of states regarding the target task using videos captured from multiple viewpoints. MVR leverages video-text similarity from a frozen pre-trained VLM to learn a state relevance function that mitigates the bias towards specific static poses inherent in image-based methods. Additionally, we introduce a state-dependent reward shaping formulation that integrates task-specific rewards and VLM-based guidance, automatically reducing the influence of VLM guidance once the desired motion pattern is achieved. We confirm the efficacy of the proposed framework with extensive experiments on challenging humanoid locomotion tasks from HumanoidBench and manipulation tasks from MetaWorld, verifying the design choices through ablation studies.
