Search papers, labs, and topics across Lattice.
This paper challenges the conventional view that interpretability in language models is a trade-off against performance by integrating interpretability as a core constraint in the training process. The authors demonstrate that as models scale, both interpretability and capability improve, with their Steerling-8B model exhibiting enhanced alignment with human-understandable concepts and more disentangled representations. Notably, this approach enables closed-loop interventions for output diagnosis and behavior correction without the need for retraining, while remaining competitive against larger models trained on significantly more compute.
Scaling language models with interpretability as a core constraint reveals that they can become more capable and understandable simultaneously, defying traditional trade-off assumptions.
Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.
