Search papers, labs, and topics across Lattice.
This paper introduces Sharpness-Aware Policy Optimization (SHAPO), a novel approach to safe exploration in reinforcement learning that leverages epistemic uncertainty by evaluating gradients at perturbed parameters. By adjusting policy updates to be pessimistic in the face of uncertainty, SHAPO amplifies the impact of rare unsafe actions while reducing the influence of already safe ones, thus promoting conservative behavior in less-explored areas. Experimental results demonstrate that SHAPO enhances both safety and task performance across various continuous-control tasks, outperforming existing methods and expanding their Pareto frontiers.
SHAPO redefines safe exploration by biasing policy updates toward caution in uncertain regions, leading to significant improvements in safety and performance.
Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sensitivity to parameter perturbations serves as a practical proxy for regions of high uncertainty. We propose Sharpness-Aware Policy Optimization (SHAPO), a sharpness-aware policy update rule that evaluates gradients at perturbed parameters, making policy updates pessimistic with respect to the actor's epistemic uncertainty. Analytically we show that this adjustment implicitly reweighs policy gradients, amplifying the influence of rare unsafe actions while tempering contributions from already safe ones, thereby biasing learning toward conservative behavior in under-explored regions. Across several continuous-control tasks, our method consistently improves both safety and task performance over existing baselines, significantly expanding their Pareto frontiers.
Mila