Search papers, labs, and topics across Lattice.
The paper introduces ROTATE, a data-free method that disentangles MLP neurons in LLMs by optimizing rotations of neuron weights to maximize kurtosis in vocabulary space. This reveals sparse, interpretable directions called vocabulary channels, which correspond to specific concepts encoded by the neuron. Experiments on Llama-3.1-8B-Instruct and Gemma-2-2B-it show that ROTATE recovers faithful vocabulary channels, enabling selective ablation of input activations and concept promotion, and yielding neuron descriptions that outperform activation-based methods by 2-3x.
Unlocking interpretability just got easier: ROTATE disentangles MLP neurons without data, revealing sparse, concept-aligned vocabulary channels directly from model weights.
Interpreting the information encoded in model weights remains a fundamental challenge in mechanistic interpretability. In this work, we introduce ROTATE (Rotation-Optimized Token Alignment in weighT spacE), a data-free method requiring no forward passes that disentangles MLP neurons directly in weight space. Our approach relies on a key statistical observation: neurons that encode coherent, monosemantic concepts exhibit high kurtosis when projected onto the model's vocabulary. By optimizing rotations of neuron weights to maximize their vocabulary-space kurtosis, our method recovers sparse, interpretable directions which we name vocabulary channels. Experiments on Llama-3.1-8B-Instruct and Gemma-2-2B-it demonstrate that ROTATE consistently recovers vocabulary channels that are faithful to the neuron's behavior. ablating individual channels selectively disables corresponding input activations or the promotion of specific concepts. Moreover, aggregating channel-level descriptions yields comprehensive neuron descriptions that outperform optimized activation-based baselines by 2-3x in head-to-head comparisons. By providing a data-free decomposition of neuron weights, ROTATE offers a scalable, fine-grained building block for interpreting LMs.
Google Research