Search papers, labs, and topics across Lattice.
This paper enhances the effectiveness of LLM auditors in alignment auditing through a reinforcement learning framework, addressing the limitations of current automated auditors in coherent investigations. By employing a policy that investigates target models with potentially hidden behaviors and utilizing an LLM judge for reward assessment, the authors demonstrate that pairwise rewards lead to more robust training outcomes. The results indicate significant improvements in investigation quality and audit realism while maintaining a low false-positive rate, with generalization of auditing capabilities across different scaffolds.
Pairwise rewards in reinforcement learning can significantly boost the robustness of LLM auditors, enhancing their ability to detect hidden model behaviors with minimal false positives.
Alignment auditing of frontier models increasingly relies on LLM auditors to surface undesirable behaviors at scale, but current automated auditors can struggle with coherent investigation and audit realism. In this work, we improve LLM auditors with reinforcement learning. In our best training environment, the policy investigates target models that potentially possess hidden behaviors planted via their system prompt. An LLM judge, which knows whether the target has a hidden behavior, holistically compares the policy's investigation to a reference investigation to determine the reward. With systematic ablations, we find that pairwise rewards yield more robust training compared to pointwise rewards, and that adding targets without planted behaviors helps maintain a low false positive rate. Training improves investigation quality against targets with planted behaviors, the rate of concerning behaviors surfaced in unmodified production models, and audit realism, while false-positive rates stay below 1%. Furthermore, auditing capabilities generalize across scaffolds: performance on AuditBench's adversarially fine-tuned targets substantially improves [Sheshadri et al., 2026].
