Search papers, labs, and topics across Lattice.
This paper explores the use of Visual Inspection of Policies (VIP) to create open-ended curricula in Reinforcement Learning by analyzing recorded episode videos of agent behavior. By employing a Video Language Model (VLM) to assess task difficulty and recommend curricula, the authors demonstrate that VIP outperforms traditional text-based methods and scalar task scores in generating effective learning tasks. The results indicate that even a lightweight VLM like VideoLLaMa2-7B can significantly enhance the training of agents in complex environments such as the StarCraft Multi-Agent Challenge.
Directly analyzing policy videos can yield more effective training curricula than traditional text-based evaluations in multi-agent reinforcement learning.
Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills. A major challenge when designing such curricula is assessing task difficulty relative to the agent's current learning progress. While previous work has explored using scalar task scores or textual summaries of the agent's behavior, here we study a different approach: directly inspecting policy behavior via recorded episode videos. We introduce a simple yet effective instantiation of this approach which leverages a Video Language Model (VLM) to both process these videos and provide curriculum recommendations, which we call Visual Inspection of Policies (VIP). Since videos can naturally contain any number of controllable agents, we empirically study VIP on the StarCraft Multi-Agent Challenge (SMAC). We show that even with a lightweight and openly accessible VLM (VideoLLaMa2-7B), VIP can use policy videos to generate more effective curricula than both its text-only ablation and methods that rely on scalar task scores.
