Search papers, labs, and topics across Lattice.
This paper investigates weight regularization techniques (L1 and L2 penalties) to improve the stability and consistency of features learned by sparse autoencoders (SAEs). They find that L2 regularization, combined with tied initialization and unit-norm decoder constraints, significantly increases cross-seed feature consistency on MNIST and improves feature sharing and steering success in TopK SAEs trained on language model activations. Furthermore, L2 regularization aligns text-based feature explanations with functional controllability, as indicated by a stronger correlation between auto-interpretability scores and steering success.
L2 weight regularization unlocks stable and steerable sparse autoencoders, doubling steering success rates and aligning feature explanations with functional controllability.
Sparse autoencoders (SAEs) are widely used to extract human-interpretable features from neural network activations, but their learned features can vary substantially across random seeds and training choices. To improve stability, we studied weight regularization by adding L1 or L2 penalties on encoder and decoder weights, and evaluate how regularization interacts with common SAE training defaults. On MNIST, we observe that L2 weight regularization produces a core of highly aligned features and, when combined with tied initialization and unit-norm decoder constraints, it dramatically increases cross-seed feature consistency. For TopK SAEs trained on language model activations (Pythia-70M-deduped), adding a small L2 weight penalty increased the fraction of features shared across three random seeds and roughly doubles steering success rates, while leaving the mean of automated interpretability scores essentially unchanged. Finally, in the regularized setting, activation steering success becomes better predicted by auto-interpretability scores, suggesting that regularization can align text-based feature explanations with functional controllability.
