Search papers, labs, and topics across Lattice.
The paper introduces a method for automatically identifying biases in reward models (RMs) used in LLM post-training. They use an LLM to iteratively propose and refine candidate biases in natural language. The method successfully recovers known biases and discovers new ones, such as Skywork-V2-8B favoring responses with redundant spacing and hallucinated content.
Skywork-V2-8B, a leading open-weight reward model, is shown to mistakenly favor responses with redundant spacing and hallucinated content, revealing critical vulnerabilities in current RM training.
Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.
