Search papers, labs, and topics across Lattice.
This study critiques the ability of sparse autoencoders (SAEs) to capture human-like category boundaries and typicality structures in language model representations, contrasting with previous findings using dense embeddings. By employing overlap over active SAE latent sets as a similarity measure, the authors demonstrate that while SAEs can recover compositional structures in controlled environments, they fail to align with human conceptual frameworks in natural text. The findings reveal a significant disconnect between human judgments of conceptual change and the SAE's internal similarity tracking, suggesting that SAEs do not operate under straightforward bag-of-features semantics in complex scenarios.
Sparse autoencoders may misrepresent human category structures, failing to capture the nuanced typicality that dense embeddings can.
Shani et al. (2026) show that LLM representations broadly recover human category boundaries, while failing to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations. We revisit their approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure. We first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. Extending the human-concepts analysis to SAE set similarities, we find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states, but instead track model-internal similarity structure. To probe this gap further, we study active latent sets under well-controlled semantic modifications, revealing a substantial mismatch between human judgements of conceptual change and change in the SAE active set. We interpret this as evidence that, outside idealised settings, SAE features do not compose via simple bag-of-features semantics.
