Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the extent to which polysemanticity metrics in large language models are confounded by lexical identity, where neurons activate for the same word form (e.g., "bank") with different meanings. Through a 2x2 factorial decomposition, they find that lexical-only activation consistently exceeds semantic-only activation across models from 110M to 70B parameters. Correcting for this confound improves word sense disambiguation and knowledge editing, suggesting that current superposition metrics overestimate true polysemanticity.
LLMs aren't as polysemantic as we thought: much of the apparent superposition is just neurons firing for the same word with different meanings.
If the same neuron activates for both"lender"and"riverside,"standard metrics attribute the overlap to superposition--the neuron must be compressing two unrelated concepts. This work explores how much of the overlap is due a lexical confound: neurons fire for a shared word form (such as"bank") rather than for two compressed concepts. A 2x2 factorial decomposition reveals that the lexical-only condition (same word, different meaning) consistently exceeds the semantic-only condition (different word, same meaning) across models spanning 110M-70B parameters. The confound carries into sparse autoencoders (18-36% of features blend senses), sits in<=1% of activation dimensions, and hurts downstream tasks: filtering it out improves word sense disambiguation and makes knowledge edits more selective (p = 0.002).