Search papers, labs, and topics across Lattice.
This paper critiques the conventional formulation of Sparse Autoencoders (SAEs) in mechanistic interpretability, revealing that their one-dimensional decoder assumption leads to feature splitting and inefficiencies. By introducing Subspace-Aware Sparse Autoencoders (SASA), which utilize learned decoder subspaces and enforce block sparsity, the authors demonstrate a significant reduction in feature fragmentation and an improvement in interpretability. The empirical results on GPT-2 and Mistral-7B show that SASA not only matches but often exceeds the performance of standard SAEs while requiring half the token budget for training.
Feature splitting in traditional Sparse Autoencoders can be mitigated by using learned subspace decoders, leading to improved interpretability and efficiency in mechanistic analysis.
Sparse Autoencoders (SAEs) are widely used for mechanistic interpretability in large language models, yet their formulation assigns each latent feature a single decoder direction, implicitly assuming features to be one-dimensional. We show that this assumption mismatches with the multi-dimensional structure of model features, provably inducing feature splitting through two distinct mechanisms. Geometrically, reconstructing a feature of intrinsic dimension $d_i \ge 2$ to error $\varepsilon$ with single-direction decoders forces a number of atoms that is exponential in $d_i$. From an end-to-end optimization perspective, this splitting is not merely possible but actively preferred. We prove that there exists a continuous path from the true $d_i$-dimensional basis to a strictly lower risk of the $\ell_1$-regularized SAE objective, whose descent directions drive any trained dictionary into that exponential regime. A single coherent feature is therefore fragmented across many near-collinear latents, producing spurious multiplicity and obscuring the intrinsic geometry. Motivated by this, we introduce Subspace-Aware Sparse Autoencoders (SASA), which replace single-vector decoders with learned decoder subspaces, enforce block sparsity via Top-$s$ group gating, and adapt each group's effective rank with a nuclear-norm regularizer. We then show that once the block size satisfies $r \ge d_i$, a single group not only can represent the entire feature slice but is the global minimizer of the SASA objective. This consolidation yields a sample complexity polynomial in $d_i$ rather than exponential -- a decisive advantage given that every training activation costs an LLM forward pass. Empirically, on GPT-2 and Mistral-7B, SASA reduces feature splitting and absorption, improves monosemanticity and interpretability, and matches or exceeds standard SAEs while training on roughly half the token budget.
