Search papers, labs, and topics across Lattice.
This paper formalizes the integration of large language models (LLMs) into reinforcement learning (RL) by framing it as a Goal-Augmented Markov Decision Process, ensuring that LLM-derived reward signals maintain optimal policy invariance. The authors demonstrate that using the LLM's per-state progress score as a bounded potential function preserves the optimal policy set, even in the presence of inaccuracies in the LLM scores. Numerical verification across various configurations, including adversarial scenarios, confirms the robustness of this approach compared to traditional LLM-as-reward methods.
LLM-derived rewards can maintain optimal policy invariance even when the feedback is inaccurate, challenging the limitations of conventional reward shaping methods.
Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.
