Search papers, labs, and topics across Lattice.
The paper introduces a causal decomposition approach to improve reward model (RM) robustness by explicitly encouraging the RM to ground preferences in the prompt's intent. A decoder is trained to map candidate answers to the latent intent embedding of the input prompt, and the reconstruction error is used as a regularization signal during RM training. Experiments on Gemma models show improved RewardBench accuracy and robustness to spurious cues like response length and sycophancy across math, helpfulness, and safety benchmarks.
Reward models can be made more robust to spurious cues like length and sycophancy by explicitly training them to understand the *intent* behind a prompt.
Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.
