Search papers, labs, and topics across Lattice.
This paper provides a mathematical framework for understanding the Jacobian lens (J-lens) as a causal transfer operator that maps intermediate activations to future readouts in language models. By analyzing the Jacobian matrix, the authors reveal its sparse causal geometry and the concentration of energy in specific pathways, leading to short-horizon and sparse concept predictions. The findings not only clarify the J-lens's role in visualizing concepts but also introduce a decoupling method that significantly improves its effectiveness in reading out correct intermediate concepts.
The J-lens reveals that language models operate with a surprisingly sparse causal structure, concentrating energy in specific pathways to predict future outputs.
The Jacobian lens (J-lens) has been proposed as a way to read verbalizable representations from language models. However, its principle and meaning lack a detailed and theoretical discussion. We provide a mathematical view of this interpretation and of its assumed causal structure. Besides treating the J-lens as a heuristic probe, we further regard it as a first-order causal transfer operator from intermediate activations to expected future readouts. We study the Jacobian matrix as the optimal local linear approximation of the downstream mapping, analyze its global approximation behavior and bias, and identify its mathematical meaning as an expectation over anticipated future readouts. Further analysis of the Jacobian energy distribution reveals that its causal geometry is highly sparse. The energy decays with depth, concentrates in an extremely small proportion, and decomposes into diagonal pathways and specific critical positions. This decomposition further resolves the expectation of the J-lens over future outputs into short-horizon and sparse concept predictions, providing a more intuitive attribution and explanation for the ability of the J-lens to visualize concepts during the thinking process. Based on the theory, we propose a simple but effective improvement strategy and decoupling method for the J-lens, which significantly enhances the ability of the J-lens to read out correct intermediate concepts.
