Search papers, labs, and topics across Lattice.
This paper introduces CircuitLasso, a scalable circuit-learning method that utilizes sparse linear regression to interpret large language models (LLMs) by learning sparse circuits over their components. The approach addresses the challenge of polysemantic neurons in LLMs, which complicate the interpretation of learned circuits, by efficiently uncovering relationships among sparse autoencoder features. Key results demonstrate that CircuitLasso achieves structural accuracy comparable to state-of-the-art methods while significantly reducing computational costs, and it successfully applies these insights to improve performance on a domain-generalization task.
CircuitLasso reveals how human-interpretable semantic features propagate through LLMs, achieving high accuracy with drastically lower computational costs.
A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior. However, raw neurons are polysemantic, making learned circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable circuit-learning approach based on sparse linear regression. CircuitLasso recovers circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.
