Search papers, labs, and topics across Lattice.
This paper introduces Conditional Co-Ablation (CoAx), a novel approach for recovering self-repair backups in transformer circuits by evaluating the conditional growth of ablation effects after primary components are removed. The method significantly improves the identification of backup components, achieving a ROC-AUC score of 0.91 on the GPT-2-small IOI circuit, compared to 0.33 with traditional methods. By revealing second-order interactions that are typically obscured in standard component-level interventions, CoAx enhances model interpretability and enables more effective capability knockout and structured pruning strategies.
Self-repair mechanisms in transformers can obscure critical components, but Conditional Co-Ablation reveals hidden backups that are essential for understanding model behavior.
Mechanistic interpretability often relies on component-level interventions to discover how a model produces a behavior. This guides attribution, capability knockout, and model pruning downstream to operate by scoring each unit by the effect of ablation in isolation. Such first-order scoring is natural when component importance is additive, but becomes misleading when a transformer self-repairs: after a primary component is removed, a dormant backup can take over, muting the primary's measured effect while the backup itself appears irrelevant on the intact model. We recast this failure as a recovery task, conditional circuit completion, and introduce Conditional Co-Ablation (CoAx), a label-free, output-grounded score that asks how much each remaining unit's ablation effect grows once a primary set has been removed. This conditional growth exposes the second-order interaction that single-unit scores discard. On the GPT-2-small IOI circuit, CoAx raises backup-head recovery from 0.33 to 0.91 ROC-AUC, outperforming all baselines, including self-repair-aware gradient scores (best 0.82); counterfactual patching verifies that the recovered heads causally carry the repair. The same label-free procedure transfers to induction across eight models. Beyond discovery, the recovered backups correct self-repair-masked attribution, identify the components required for capability knockout, and yield repair-aware structured pruning scaling from 124M to 7B. Component importance is therefore not merely an isolated-unit property: in robust circuits, the components that matter can become visible only under the interventions that make them necessary.
