Search papers, labs, and topics across Lattice.
This paper introduces CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel framework that evaluates explanations of language model behaviors by assessing their predictive power on counterfactual inputs. The study reveals that commonly used interpretability techniques do not enhance an agent's ability to predict counterfactual behaviors, challenging existing assumptions in the field. Additionally, the authors demonstrate that training models on CHIVE-generated counterfactual data improves generalization across out-of-distribution scenarios, highlighting the framework's utility in generating high-quality explanations for LLM behaviors.
Counterfactual experiments reveal that traditional LLM interpretability techniques fail to improve predictive accuracy, challenging long-held beliefs in the field.
Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.
