Search papers, labs, and topics across Lattice.
This paper introduces Certified Interventional Fidelity (CIF), a novel statistical framework for evaluating causal claims in mechanistic interpretability by providing confidence intervals and anytime-valid confidence sequences. CIF addresses the limitations of traditional point estimates by framing evaluations as causal estimands and allowing for adaptive intervention sampling, significantly reducing certification costs by 10-30x. The method is validated through experiments on MNIST abstractions and GPT-2 Small IOI circuits, demonstrating its ability to certify high-fidelity claims and clarify the statistical support for method differences.
CIF reveals that many purported causal claims in mechanistic interpretability lack statistical support, challenging the reliability of existing evaluation methods.
Mechanistic interpretability often evaluates explanations by intervening on a model: swapping hidden states, patching activations, ablating components, or comparing a compressed model to the original one. These experiments are usually summarized by a point estimate, even though the evaluation may be monitored while it runs or adapted toward suspected failures. This makes it hard to tell whether a reported fidelity or patching effect is a stable causal claim or a consequence of finite sampling and evaluation choices. We introduce Certified Interventional Fidelity (CIF), a statistical layer for interventional interpretability evaluations. CIF first writes the quantity being reported as a causal estimand: an expectation of a bounded score over a stated input distribution and a stated intervention distribution. It then provides confidence intervals and anytime-valid confidence sequences for this estimand, including under adaptive intervention sampling via bounded mixture importance weighting. We instantiate CIF with Hoeffding-style sequences and variance-adaptive betting sequences, the latter reducing certification cost by 10-30x in our experiments. On MNIST abstractions and GPT-2 Small IOI circuits, CIF certifies high-fidelity claims, shows when apparent method differences are not statistically supported, and makes sensitivity to the intervention distribution explicit.
