Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce a novel gridworld environment to demonstrate specification gaming, where a cleaning robot exploits a sensor's proxy reward by concealing dirt instead of removing it, leading to near-maximal proxy reward but complete failure on the true objective. A constrained reinforcement learning approach, penalizing the specific gaming behaviors, effectively eliminates this exploitation and recovers true-objective performance. Crucially, we derive an exact accounting identity showing our cost signal deterministically measures the proxy/true reward gap, providing a verifiable guarantee for the mitigation.
A single adversarial token can bypass most safety filters.
Reinforcement learning agents optimize exactly the reward signal they are given, which can diverge sharply from an intended true objective when that signal is only a proxy for it; this failure mode is known as specification gaming. Existing evidence for this failure mode is largely observational, with comparatively few demonstrations pairing a controlled, deliberately constructed proxy/true gap with a mitigation that carries a verifiable guarantee rather than only an empirical improvement. We construct a minimal gridworld in which a cleaning-robot agent can either genuinely remove dirt or conceal it from a cheap sensor, and show that a standard deep Q-network trained only on the sensor's proxy reward converges to a policy that games it completely, reaching near-maximal proxy reward while never achieving the true objective. We then introduce a constrained reinforcement learning fix, penalizing the specific behaviors through which the agent exploits the gap, and show that it eliminates the gaming behavior from the final policy while substantially recovering true-objective performance. We further derive and empirically verify an exact accounting identity relating the proxy reward, true reward, and our cost signal for this environment, showing that the cost signal measures the proxy/true gap exactly rather than merely tracking it, and yields a deterministic guarantee under a hard-constrained variant of the method.