Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper posits that the observed "self-repair" in language models following component ablation is not a novel emergent property but rather the predictable behavior of pre-existing counterweight components. The authors demonstrate that the response of a fine-grained unit $r$ to an intervention $\lambda$ follows an affine law, $E_r(\lambda)=\mathrm{own}_r+\gamma_r\lambda$, where $\gamma_r$ is a fixed coefficient indicating whether the unit counteracts or reinforces the removed signal. This framework successfully explains observed compensatory behaviors across multiple model families and identifies specific components acting as counterweights.
Language model "self-repair" is a predictable consequence of pre-existing counterweight mechanisms, not emergent compensation, governed by an affine law.
Ablate a component of a language model, and other components often appear to adjust and compensate. This phenomenon, termed self-repair, has been observed repeatedly, but its mechanism remains unclear. The most systematic study to date concluded that self-repair is noisy and unlikely to have a single explanation. We argue that it has one: a gain already present before any ablation. Any intervention on a causally important component can be viewed as a point on a coordinate axis $\lambda$, the signed strength of a counterfactual contrast. Hence, conventional ablation methods are uncalibrated points on this axis. We show that the causal repair response for a fine-grained unit $r$ is governed by an affine law, $E_r(\lambda)=\mathrm{own}_r+\gamma_r\lambda$. The slope $\gamma_r$ is a fixed coefficient that consistently influences the model, with or without ablation, and its sign determines whether the unit counteracts or reinforces the removed signal. On a factual-verdict task across four models from distinct families (Gemma, Qwen, LLaMA, and Mistral), we identify components including MLP neurons, OV neurons, and singular directions that follow this affine law, 68 of 81 downstream directions in all. Moreover, we can anticipate the magnitude of $\gamma_r$ from the fixed weights. On the IOI circuit of GPT-2 Small, seven of the ten heads the intervention can reach follow the law, and all seven are counterweights. From this perspective, what may appear as self-repair is a counterweight performing its usual operation when the contrastive signal emerges at the core.