Search papers, labs, and topics across Lattice.
This paper trains a Transformer to perform base-digit extraction (finding the coefficient of B^D in the base-B expansion of N) and investigates whether the model internally computes the closed-form solution's intermediate values. While linear probes suggest the model represents these intermediate values, causal interventions reveal that the identified localized causal route does not transmit them to the output stream, indicating a disconnect between representation and computation. The study highlights the potential for probe-based analyses to mislead about actual algorithmic processing within neural networks.
Even with a clear algorithmic solution available, Transformers can represent intermediate values without actually using them in the way probes suggest, challenging the reliability of probe-based mechanistic interpretability.
Structured prompts require integrating components according to task-relevant relations. How a network implements this integration is often hard to judge in language or vision, where those relations are rarely specified precisely enough to define a candidate internal algorithm. Arithmetic offers a cleaner setting. We study a Transformer trained on base-digit extraction: given $N$, $B$, and $D$, it must report the coefficient of $B^D$ in the base-$B$ expansion of $N$. The closed-form solution, $\lfloor N/B^D \rfloor \bmod B$, provides explicit candidate algorithmic intermediates. Across three seeds, the model reaches 99.83% exact-answer accuracy on held-out number-base intersections, establishing reliable task competence. Linear probes decode the intermediates, making staged arithmetic computation plausible. Causal tests then separate representation from use: within the localized route from the stream with $D$ as input to the output positions, behavior depends on early $D$-selective communication, independent of $N$ and $B$. Relatedly, a sparse circuit search finds mostly separate $N$, $B$, and $D$ routes that combine late rather than the staged route suggested by the probes. Thus, the model represents the intermediates that make the closed-form solution plausible, but the identified localized causal route does not transmit them to the output stream. This case shows that probe-based conclusions can diverge sharply from causal observations, even when explicit algorithmic hypotheses are available.
