Search papers, labs, and topics across Lattice.
This paper introduces Occupancy Reward Shaping (ORS), a method that utilizes generative world models to improve credit assignment in offline goal-conditioned reinforcement learning by extracting temporal information from the learned occupancy measure. By employing optimal transport, ORS transforms this temporal geometry into a reward function that effectively captures goal-reaching dynamics, significantly addressing the challenges posed by sparse rewards. The method not only maintains the optimal policy but also achieves a remarkable 2.2x performance improvement across 13 diverse long-horizon tasks, including real-world applications in nuclear fusion control.
Extracting temporal geometry from generative models can boost reinforcement learning performance by over 2x without changing the optimal policy.
The temporal lag between actions and their long-term consequences makes credit assignment a challenge when learning goal-directed behaviors from data. Generative world models capture the distribution of future states an agent may visit, indicating that they have captured temporal information. How can that temporal information be extracted to perform credit assignment? In this paper, we formalize how the temporal information stored in world models encodes the underlying geometry of the world. Leveraging optimal transport, we extract this geometry from a learned model of the occupancy measure into a reward function that captures goal-reaching information. Our resulting method, Occupancy Reward Shaping, largely mitigates the problem of credit assignment in sparse reward settings. ORS provably does not alter the optimal policy, yet empirically improves performance by 2.2x across 13 diverse long-horizon locomotion and manipulation tasks. Moreover, we demonstrate the effectiveness of ORS in the real world for controlling nuclear fusion on 3 Tokamak control tasks. Code: https://github.com/aravindvenu7/occupancy_reward_shaping; Website: https://aravindvenu7.github.io/website/ors/
CMU ML