Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to explainability in Large Language Models (LLMs) by employing causal graphs to model LLM inference processes. By utilizing a four-phase method that includes an MCMC-inspired counterfactual augmentation procedure, the authors enhance the interpretability of LLM outputs through the discovery of class-discriminative concepts and their mapping to LLM-perceived states. The results demonstrate that the constructed causal graphs effectively capture meaningful dependencies and provide a transparent view of LLM reasoning across various tasks, including disease diagnosis and sentiment analysis.
Causal graphs reveal how LLMs perceive concepts, transforming our understanding of model reasoning and interpretability.
Causal graphs provide a high-level language for making mechanisms transparent. Recent work uses Large Language Models (LLMs) to recover causal graphs of external-world processes. Instead, in this paper, we use causal graphs to model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce an MCMC-inspired counterfactual augmentation procedure that expands the sparse observational data through chains of counterfactuals. This enables stable causal discovery with $\sigma$-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs for predictive fidelity and structural stability, and the MCMC-inspired augmentation for convergence and downstream utility. Our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs'reasoning. Together, this paper provides a foundation for concept-level explainability of LLMs.
