Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of clustering co-activation statistics in identifying attention-head circuits within neural networks, specifically using a sparse-autoencoder approach validated through causal ablation. The researchers found that while communities of attention heads can be identified and show closure in dense models, this method fails to confirm circuit validity in a Mixture-of-Experts model, where ablation unexpectedly improves performance. The results emphasize that while co-activation can suggest potential circuits, true validation requires rigorous closure testing to avoid misleading conclusions about functionality.
Clustering co-activation statistics may hint at attention-head circuits, but only closure testing can confirm their validity—leading to surprising findings in Mixture-of-Experts models.
Interpretability increasingly treats groups of components, not individual units, as the basic object, and proposes to find them by clustering co-activation statistics. We ask whether such a cheap signal actually identifies an attention-head circuit. Adapting a sparse-autoencoder clustering recipe to attention heads -- but validating by causal ablation rather than reconstruction -- we cluster heads and then run a closure test: ablate the discovered community and compare per-example damage to matched-random controls. Across two dense 1B-scale models (Pythia 1B, OLMo 1B) and two input distributions, the communities pass closure. In a Mixture-of-Experts model (OLMoE-1B-7B), route-conditional clustering recovers a statistically real signal that nonetheless does not survive closure -- ablation improves loss, the wrong direction. Extending closure across training, attention-target selectivity and participation ratio decouple from function in both directions. We conclude that a cheap signal is a circuit proposal, not a confirmed circuit; closure is what separates them.