Search papers, labs, and topics across Lattice.
The paper introduces a Bayesian mitigation strategy, termed "Golden Handcuffs," to improve the safety of reinforcement learning agents by expanding the agent's subjective reward range to include a large negative value, making the agent risk-averse to potentially unsafe novel strategies. They prove that this approach achieves both capability (sublinear regret against the best mentor with mentor-guided exploration) and safety (no low-complexity predicate is triggered by the optimizing policy before it is triggered by a mentor). The method also incorporates a simple override mechanism that yields control to a safe mentor when the predicted value drops below a threshold.
By making RL agents fear a large, subjectively possible negative reward, "Golden Handcuffs" aligns them to safer behavior without sacrificing capability.
Reinforcement learners can attain high reward through novel unintended strategies. We study a Bayesian mitigation for general environments: we expand the agent's subjective reward range to include a large negative value $-L$, while the true environment's rewards lie in $[0,1]$. After observing consistently high rewards, the Bayesian policy becomes risk-averse to novel schemes that plausibly lead to $-L$. We design a simple override mechanism that yields control to a safe mentor whenever the predicted value drops below a fixed threshold. We prove two properties of the resulting agent: (i) Capability: using mentor-guided exploration with vanishing frequency, the agent attains sublinear regret against its best mentor. (ii) Safety: no decidable low-complexity predicate is triggered by the optimizing policy before it is triggered by a mentor.
