Search papers, labs, and topics across Lattice.
This work audits automated simulatability frameworks, such as ConSim, which replace costly human evaluations by using LLMs to judge explanation quality based on how accurately they predict a model's outputs. The authors reveal that LLM simulators routinely bypass explanations entirely, achieving high simulatability either by solving the downstream classification task directly via semantic class names or by exploiting explanations that leak anonymized label mappings. Across diverse datasets, explanation families, and simulator architectures, simulator predictions are primarily driven by underlying task priors rather than the explanations themselves, invalidating common automated interpretability rankings.
Automated benchmarks for explainability are fundamentally brittle: LLM "simulators" consistently game the metric by solving the task directly through semantic priors or exploiting label leakage rather than actually relying on the explanations.
Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs. Since human evaluation is costly, automated simulatability replaces human explainees with LLM simulators, as proposed in ConSim (Poch茅 et al., 2025) for large-scale experiments. We qualitatively replicate and extend ConSim's ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs, and identify two limitations. First, when class names are meaningful, simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations. Second, class anonymization can reward explanations for leaking the hidden label mapping, a limitation we expose with a new classes-as-concepts baseline. These results are consistent with a shortcut hypothesis: in the tested settings, simulator predictions mainly rely on task priors, while explanations produce small changes. We derive recommendations for more robust automated simulatability evaluations.
