Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Expander Sparse Autoencoders (SAEs), which optimize the parameter efficiency of sparse autoencoders by employing a left-$d$-regular expander mask, allowing for a significant reduction in the number of learned decoder values while maintaining the integrity of the sparse-coding problem. The experiments demonstrate that this approach achieves a remarkable storage-fidelity tradeoff, with the Qwen2.5-3B model using 293 times fewer decoder values than a full dense decoder while retaining 84% of the performance in terms of cross-entropy loss. The theoretical contributions include conditions for identifiability of sparse codes, enhancing the understanding of the relationship between expansion, column flatness, and recovery of support in compressed sensing contexts.
Achieving 293 times fewer decoder values without sacrificing performance could revolutionize the efficiency of neural network interpretability methods.
Sparse autoencoders (SAEs) decompose internal activations of neural networks into sparse linear combinations of learned features by fitting an overcomplete dictionary $\mathbf{W}\in\mathbb{R}^{m\times n}$ with $m<n$, and inferring a sparse code $\mathbf{x}\in\mathbb{R}^n$ from $\mathbf{h}\approx\mathbf{W}\mathbf{x}$. This inference problem closely resembles the canonical setup of compressed sensing, but dense decoders requires $O(mn)$ learned values, which becomes costly at large feature counts. We introduce Expander SAEs: TopK SAEs whose decoder and tied encoder are supported on a left-$d$-regular expander mask with $d\ll m$, learning only $dn$ decoder values while keeping the sparse-coding problem $(m,n,k)$ fixed. The same structure reduces storage and turns the matching-pursuit correlation step $\mathbf{W}^\top \mathbf{r}$ in OMP into an $O(dn)$ gather-and-reduce operation. Our experiments show that across Pythia-70M/160M, Qwen2.5-3B, and Llama-3.2-1B residual-stream activations, varying $d$ traces a consistent storage--fidelity frontier, and that at the most compressed modern-LM setting, Qwen2.5-3B with $d=7$ uses $293\times$ fewer learned decoder values than the full dense decoder while retaining $84$% of dense CE-loss recovered. Control experiments show that the improved storage--fidelity tradeoff is driven by sparse, diverse decoder support structure rather than by fewer learned decoder values, and that when sparse and dense decoders are compared at matched parameter count, part of the remaining gap comes from encoder amortisation. On the theoretical side, we show that expansion and column flatness are sufficient for identifiability of noiseless $k$-sparse codes, and we derive complementary sufficient conditions under which OMP recovers the support exactly.