Search papers, labs, and topics across Lattice.
This paper introduces Sparse Readout Prism (SRP), a method that independently analyzes the readout structure of language models by decomposing logit outputs into contributions from sparse readout features, rather than relying on fitting corpora. The authors demonstrate that traditional lens methods can yield corpus-dependent results, while SRP reveals stable dominant features across different contexts and layers. By reconstructing logit differences with SRP, they achieve a significant improvement in accuracy over existing methods, highlighting the importance of readout features in understanding model behavior.
Traditional lens methods can mislead due to corpus dependence, but Sparse Readout Prism reveals stable readout features that enhance interpretability and accuracy in language model analysis.
A language model's prediction of its next token develops across layers, and lens methods track this process by decoding intermediate hidden states into tokens. But a lens reading reflects both the hidden state and the readout (the unembedding matrix) used to decode it. Many lenses are fit on a corpus, and we show that two lenses differing only in their fitting corpus can report different tokens for the same hidden states. We call this dependence corpus conditionality. To examine readout structure independently of the fitting corpus, we introduce Sparse Readout Prism (SRP), which decomposes the readout using only its weights and expresses any token logit or logit difference as a sum of contributions from sparse readout features. This reveals readout features as a new unit of analysis for lens readings, exposing structure that token identities can obscure and enabling comparisons across tokens, contexts, layers, and lenses. Replacing the original readout with SRP's sparse approximation reconstructs 8.9-17.3 percentage points more of the tested logit differences than the strongest of six baselines built on geometric relations among readout rows. Ablating features shifts logit differences in proportion to their SRP contributions. Although token readings vary with the fitting corpus, the dominant readout feature remains stable. Because SRP uses no corpus in its construction, it provides a control independent of the fitting corpus for lens analyses.
