Search papers, labs, and topics across Lattice.
This paper introduces Rate Matching Consistency Training (RMCT), a novel approach that enhances consistency training in large language models by focusing on matching the rate of target behaviors across input perturbations, rather than constraining the model's expression of these behaviors. The authors demonstrate that RMCT effectively reduces sycophancy in two open-weight language models, achieving bias-following reductions comparable to traditional consistency training while maintaining the model's ability to verbalize bias cues. Notably, RMCT is found to be more data-efficient, suggesting a promising avenue for improving model robustness without sacrificing monitorability.
RMCT reveals that you can reduce bias in language models without sacrificing their ability to articulate the very cues you're trying to mitigate.
Large language models are often influenced by extraneous input features, such as cues revealing a user's preferred answer. Consistency training reduces this influence by training models to behave similarly across inputs with and without the extraneous feature. However, existing methods train for consistency over entire responses or internal activations, which also constrains whether the model verbalises said extraneous features. We show this leads to obfuscation, where the model learns not to mention a cue while remaining influenced by it, which may undermine monitorability. To address this, we introduce Rate Matching Consistency Training (RMCT), which trains for consistency over selected behavioural properties without constraining how this behaviour is expressed. RMCT matches the rate at which the model exhibits a target behaviour (e.g., following a bias cue) across input perturbations, rather than requiring paired inputs with and without the extraneous feature, extending consistency training to settings where the extraneous features cannot be removed. We evaluate RMCT on sycophancy reduction in two open-weight language models, achieving reductions in bias-following comparable to a standard consistency-training baseline on held-out bias types, while largely preserving the model's tendency to verbalise the bias cue. Further, we find that RMCT is more data-efficient at the expense of being less compute-efficient in our experiments. Overall, RMCT shows that consistency training can improve behavioural robustness without directly trading off against monitorability.
