Search papers, labs, and topics across Lattice.
This paper formalizes the idea that LLMs externalize reasoning due to the Transformer architecture's inherent need for interpretable intermediate steps for sufficiently long serial cognition. They introduce "opaque serial depth" to quantify the longest computation possible without such interpretable steps. The authors then compute upper bounds on this depth for Gemma 3 models and other architectures, finding that Mixture-of-Experts models likely have lower depth than dense models.
Mixture-of-Experts models might be hiding more of their reasoning than we thought, thanks to a newly quantified "opaque serial depth" metric.
Large language models (LLMs) tend to externalize their reasoning in their chain of thought, making the chain of thought a good target for monitoring. This is partially an inherent feature of the Transformer architecture: sufficiently long serial cognition must pass through the chain of thought (Korbak et al., 2025). We formalize this argument through the notion of opaque serial depth, given by the length of the longest computation that can be done without the use of interpretable intermediate steps like chain of thought. Given this formalization, we compute numeric upper bounds on the opaque serial depth of Gemma 3 models, as well as asymptotic results for additional architectures beyond standard LLMs. We also open-source an automated method that can calculate upper bounds on the opaque serial depth of arbitrary neural networks, and use it to demonstrate that Mixture-of-Experts models likely have lower depth than dense models. Overall, our results suggest that opaque serial depth is a useful tool for understanding the potential for models to do significant reasoning that is not externalized.
DeepMind