Search papers, labs, and topics across Lattice.
This paper investigates reward hacking in RL fine-tuning of LMs by analyzing the geometry of parameter updates, finding that hacking episodes exhibit larger directional changes in parameter space compared to intended learning. To mitigate this, they introduce "trusted-direction projection," which constrains gradients during RL to remain within a subspace defined by a clean, non-hacking reference trajectory. Experiments on mathematical reasoning tasks demonstrate that this projection delays reward hacking and improves task performance.
Reward hacking isn't just about incentives, it's about wild directional swings in your model's parameter space – and constraining those swings can keep your LM on the straight and narrow.
Reward hacking arises when a model improves a proxy reward by exploiting shortcuts rather than solving the intended task. We study this failure mode through the geometry of reinforcement learning updates in language models and argue that hacking emerges when optimization drifts away from a stable low-dimensional learning trajectory. We analyze this drift through dominant singular directions of parameter updates and show that reward-hacking runs exhibit substantially larger directional change than clean runs. Motivated by this observation, we introduce trusted-direction projection, which constrains gradients to remain within a clean reference subspace. Across reward-hacking experiments on mathematical reasoning, the proposed approach delays shortcut exploitation and better preserves task performance.
