Search papers, labs, and topics across Lattice.
This paper introduces CHERRL, a controllable environment designed to reproduce and analyze reward hacking in rubric-based reinforcement learning (RL) by injecting known biases into an LLM-as-a-Judge (LaaJ). The authors demonstrate that reward hacking behaviors are often subtle and intertwined with multiple biases, complicating detection and mitigation efforts. By providing a stable testbed for studying these mechanisms, the research reveals critical insights into the discoverability and exploitability of judge biases, along with a system for automatically detecting reward hacking onset from training logs.
Reward hacking in rubric-based RL is not just common; it can be systematically reproduced and analyzed using the new CHERRL environment, revealing hidden biases that could compromise training integrity.
Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards. However, policy models may exploit latent biases in the judge, leading to reward hacking and ineffective or unsafe training outcomes. In real-world rubric-based RL, such hacking behaviors are often subtle and entangled with multiple judge biases, making them difficult to analyze, detect, and mitigate. In this paper, we introduce CHERRL, a controllable hacking environment for rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an agent-based system for automatically detecting reward hacking onset from training logs. The code and environment are publicly available at https://github.com/THUAIS-Lab/CHERRL.
Tsinghua AI