Search papers, labs, and topics across Lattice.
This paper introduces a data-centric post-training pipeline that leverages interpretability protocols to analyze preference datasets, allowing for the identification of latent concepts that dictate model behavior. By transforming the optimization of scalar rewards into a more transparent process, the authors demonstrate how to mitigate undesirable behaviors like over-stylization and sycophancy while enhancing desired traits such as safeguards and model personality. The empirical results reveal that this approach not only diagnoses problematic signals in existing datasets but also provides a framework for fine-grained user feedback to shape model learning more effectively.
Transforming post-training from opaque reward optimization into a transparent process of auditing and sculpting the learning signal could revolutionize how we guide model behavior.
Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy. To address this problem, we ask: can we inspect a preference dataset before optimization and decide, at the level of concepts, which behaviors a model should be allowed to learn? Motivated by this, we introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations, making them explicit for fine-grained user feedback. Building on this view, we unify several interpretability-based training protocols as ways of shaping rewards via feature or data interventions. Empirically, we show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality. More broadly, our results suggest that interpretability can turn post-training from optimizing opaque proxy rewards into a process of auditing and sculpting the learning signal itself.
