Search papers, labs, and topics across Lattice.
This paper identifies the $\ell_2$ norm of hidden states in Large Language Models (LLMs) as a reliable endogenous signal for assessing reasoning dynamics across model layers. By employing Sparse Autoencoders (SAEs) to probe internal reasoning features, the authors reveal that significant increases in reasoning activations occur in the later layers of the models. The findings establish a formal relationship between reasoning intensity and latent geometry, leading to the development of three innovative test-time scaling techniques that enhance reasoning performance without requiring additional training or data.
The $\ell_2$ norm of hidden states serves as a powerful indicator of reasoning intensity in LLMs, enabling new techniques that boost reasoning performance without extra training.
Recent work has sought to understand Large Language Models (LLMs) reasoning, yet a principled, model-intrinsic signal that captures its layer-wise reasoning dynamics remains underexplored. We bridge this gap by demonstrating that the l2 norm of hidden states serves as an endogenous signal of the model's reasoning intensity. Using Sparse Autoencoders (SAEs) as a diagnostic probe, we observe that LLMs' internal reasoning is marked by a sharp increase in reasoning feature activations concentrated in late layers. Motivated by this pattern, we establish a formal link between reasoning intensity and the model's latent geometry and theoretically prove that the l2 norm of hidden states bounds the activation strength of SAE reasoning features. Empirical correlation analysis and causal interventions further validate the l2 norm as a faithful indicator, where heightened norms consistently correspond to critical reasoning steps. We then introduce three test-time scaling techniques guided by l2 norms: (i) Adaptive Layer-wise Reasoning Recursion, (ii) Endogenous Reasoning State Steering, and (iii) l2-guided Response Selection, which requires no additional training or data and is compatible with advanced inference engines. Experiments across model architectures and benchmarks show that l2-norm-based techniques significantly improve reasoning performance, offering a principled yet simple lens to perceive and control LLM latent reasoning dynamics. Our code is available at https://github.com/zjy1298/The-Tell-Tale-Norm.
