Search papers, labs, and topics across Lattice.
To uncover how Vision Transformers (ViTs) internally organize visual reasoning, this work applies Cross-Layer Transcoders (CLTs) to extract both static, weight-level "global" concept circuits and dynamic, input-conditioned "instance" circuits. This dual-graph formulation provides a mechanistic framework to audit model world knowledge and isolate causal pathways used for specific visual inferences. Leveraging these circuits enables automatic discovery and surgical removal of dataset shortcuts鈥攜ielding an 11.0% performance gain over existing debiasing methods on Waterbirds鈥攚hile exposing how supervision objectives like DINO versus CLIP fundamentally alter internal circuit topologies.
ViT weights harbor fully legible, input-invariant concept graphs of the world, unlocking targeted circuit interventions that improve shortcut removal by 11.0% over prior baselines.
Vision transformers (ViTs) have achieved remarkable generalization across visual domains, yet little is known about how they internally represent the structure of the world. To address this gap, we use Cross-Layer Transcoders (CLTs) to read concept circuits from ViTs: directed graphs whose nodes correspond to sparse, interpretable concepts and edges capture concept interactions across layers. Our method yields two complementary views of model behavior. The global concept circuit is input-invariant and can be recovered directly from learned cross-layer weights, exposing the reusable "world knowledge" encoded in the model. The instance concept circuit is input-dependent and identifies the concepts and pathways actually used for a specific prediction, enabling faithful example-level explanations. We demonstrate the utility of concept circuits in three ways: (1) Automatic spurious correlation discovery: leveraging the statistics of our global concept circuits to identify shortcut dependencies within the model. (2) Spurious correlation removal: intervening on the instance concept circuit to steer the model towards correct predictions. Empirical results show that our method outperforms existing counterparts by 11.0% on the Waterbird dataset. (3) Model comparison: contrasting the global concept circuits of different foundation models (e.g., CLIP vs. DINO) to reveal how supervision paradigms shape representational structure. Our code is available at https://github.com/deep-real/VisionCLT
