Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of training language models (LMs) to generate explanations for their predictions, focusing on the phenomenon of "introspective coupling," where models produce explanations that reflect their current behaviors rather than those of their training targets. By using fixed counterfactual explanations from earlier checkpoints or similar models, the researchers found that LMs can maintain a correlation between explanations and behaviors, enabling them to adapt their introspections even as their behaviors evolve. The findings reveal that this introspective coupling can effectively track behavioral changes across various tasks, suggesting a scalable approach to enhancing model interpretability without the need for continuous supervision.
Fixed counterfactual explanations can lead LMs to generate more accurate introspections about their behaviors, even as those behaviors change over time.
When does training language models (LMs) to generate explanations of their predictions yield faithful introspection, rather than superficial imitation? We study LMs trained to explain which features of their inputs influenced their behavior, using models' counterfactual behavior on modified inputs as supervision. Surprisingly, we find that LMs trained on fixed counterfactual explanations derived from earlier checkpoints of themselves, or even from behaviorally similar models in different families, frequently produce explanations more faithful to their own current behaviors than to those of their training targets. This "introspective" coupling between LM explanations and behaviors occurs when training explanations remain sufficiently correlated with current behaviors over the course of training, even as behaviors themselves shift. We also show that introspective coupling tracks behavior shifts: when explanation training is provided concurrently with other post-training objectives, explanations track those shifts without requiring updated supervision. This phenomenon appears in multiple tasks, including sycophancy and refusal, and is robust to label noise. Overall, our results show that even fixed datasets of counterfactual explanations can provide scalable and generalizable post-training signal for introspection.
MIT CSAIL