Search papers, labs, and topics across Lattice.
This paper introduces robust general-utility reinforcement learning (RL), which addresses the robustness gap arising from utility misspecification during deployment. By employing a minimax learning framework, the authors train policies against potential deviations in the utility function, thereby generalizing existing RL frameworks and ensuring better performance in real-world applications. The proposed algorithms demonstrate convergence in both concave and nonconcave utility scenarios, with empirical validation on tasks related to LLM safety alignment and exploration maximization.
Utility misspecification can lead to significant performance drops in RL, but this new framework ensures robustness against such deviations, enhancing real-world applicability.
Reinforcement learning (RL) with general utility extends classic RL by optimizing an arbitrary utility functional of the policy-induced occupancy measure, thereby enabling a broader range of applications. However, previous work on general utility RL typically assumes the evaluation utility is fixed and correctly specified. In practice, the utility used at deployment can deviate from the training one, creating a robustness gap that prior work does not address. Motivated by this, we propose robust general-utility RL, a minimax learning framework that trains policies against utility misspecification within a prescribed uncertainty set. Our framework strictly generalizes standard general-utility RL while also providing a unified view of many existing RL frameworks, including reward-robust RL and constrained RL, through appropriate choices of the utility uncertainty set. We further develop provably convergent stochastic algorithms for two regimes. For concave utilities, we develop a projected stochastic gradient descent-ascent method and establish stationarity guarantees. For the more challenging nonconcave regime, we propose a stochastic prox-extragradient algorithm that mitigates ill-posed behavior induced by nonconcavity, with convergence guarantees to approximate first-order stationarity. Experiments on LLM safety alignment and exploration maximization tasks further corroborate the convergence behavior consistent with our theory.
