Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Circuit Condensation, a method that post-trains models to distill behaviors into smaller causal graphs by pruning low-attribution edges and employing low-rank adapters to maintain performance. The approach demonstrates that condensed circuits are significantly smaller than traditional frozen baselines, achieving an average reduction of 8.1 times across various behaviors and models. Notably, the findings reveal that weight updates during the condensation process are crucial for achieving these reductions, as opposed to merely searching for smaller circuits without updates.
Condensed causal circuits can be up to 316 times smaller than traditional frozen models while preserving task performance, revealing the power of targeted post-training in mechanistic interpretability.
One approach to mechanistic interpretability explains behavior through circuits: the components and connections that carry it. Frozen discovery often returns hundreds of edges, making them hard to inspect, compare, or verify exhaustively. We introduce Circuit Condensation, which post-trains models to concentrate behaviors into smaller causal graphs. Each round prunes low-attribution edges and trains a low-rank adapter to match the original through what remains, retaining the cut only if task performance and general capability survive. Across four behaviors and eight models, condensed circuits are smaller than the strongest frozen baseline in 30 of 32 settings, by $8.1\times$ on average and up to $316\times$. Repeating the search without weight updates produces larger circuits in 29 of 32 settings, showing that weight updates, rather than search alone, drive the reduction. Testing every subset of 19 circuits finds 11 that cannot be reduced and reveals removable edges in the rest. Pair ablations expose dependencies between edges, showing that their effects cannot be understood independently. On indirect object identification, condensation isolates 24 heads, 17 of them with documented roles, against 61 heads and 36 undocumented ones for the matched frozen circuit: a sufficient sub-circuit of the published mechanism rather than a reconstruction of it. The resulting circuit tracks the original model's next-token distribution and predicts its errors.