Search papers, labs, and topics across Lattice.
This paper introduces ProDVI, a novel framework that utilizes large language models to generate programmatic dynamics priors for initializing value networks in deep reinforcement learning (RL). By prompting a code-generating model to create executable Python functions that encapsulate coarse hypotheses about environment dynamics, ProDVI generates synthetic transitions that enhance the state-action encoder's representation before online learning begins. Experimental results demonstrate that this approach significantly improves sample efficiency in model-free RL algorithms, addressing a critical limitation in traditional RL initialization methods.
Leveraging large language models to generate synthetic transitions, ProDVI boosts sample efficiency in deep reinforcement learning without the need for extensive pre-collected data or simulators.
Deep Reinforcement Learning (RL) is notoriously sample inefficient. One contributing factor is that RL agents are typically initialized from scratch, forcing them to acquire task-relevant knowledge through online interaction. Existing approaches obtain informative initializations through pre-collected datasets, high-fidelity simulators, or meta-learning over related tasks, but these prerequisites may be difficult to access or even unavailable. In this paper, we propose Programmatic Dynamics Priors for Value Network Initialization (ProDVI), a framework that leverages the commonsense and domain knowledge encoded in large language models to initialize RL agents without relying on these resources. Specifically, ProDVI prompts a code-generating language model to produce executable Python functions that encode coarse hypotheses about environment dynamics. These functions are then used to generate synthetic transitions. Based on these transitions, we construct an auxiliary dynamics prediction objective to pretrain the state-action encoder of the value network in an actor-critic framework. The learned representation provides dynamics-aware inductive biases before online RL begins. Importantly, the generated programs are used only for representation pretraining and are not required to faithfully simulate the target environment. While the generated programs may be inaccurate, their induced initialization can be corrected through online learning from real transitions and rewards. Experiments on OpenAI Gym and DeepMind Control Suite tasks show that ProDVI can effectively improve the sample efficiency of model-free RL algorithms.
