Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Looped Transformers, employing shared weights across recurrent layers, exhibit strong performance on iterative tasks by potentially reusing learned algorithms. This work investigates whether each loop indeed performs the same computation, finding that progress is not fixed per loop and that a learned linear layer $J$ steers computations by modifying the entering hidden state. Activation patching reveals attention patterns as a causal pathway for this routing, with intermediate supervision during training influencing the transitions $J$ can induce, suggesting $J$ selects learned computations rather than creating new ones.
A single learned linear layer, not the shared Transformer weights, dictates distinct computations within each recurrent loop of a Looped Transformer, with attention patterns causally routing these operations.
Looped Transformers repeatedly apply the same set of Transformer layers, giving them a recurrent architecture for latent computation. Their strong performance on iterative reasoning and length-generalization tasks suggests an appealing explanation: recurrence may provide an inductive bias that lets the model reuse a learned algorithm across loops. However, weight sharing alone does not imply that every loop performs the same operation. This raises a basic question: is each loop actually repeating the same computation, and if not, what routes the shared parameters to different operations? We study this question using graph walks as a test case. In the model's native trajectories, decoded predictions can advance by different numbers of graph steps or remain at a reached target, showing that recurrent progress need not follow a fixed one-loop-one-step pattern. We then show that a frozen loop can be steered toward different transitions by modifying its entering hidden state: a learned linear layer $J$ selects the desired transition without changing the shared Transformer layers. To test how this steering works, we use activation patching and find that attention patterns can recover its effects and switch the selected transition. Across five matched pairs of graph models, changing intermediate supervision during backbone training changes which transitions $J$ can induce. This suggests that $J$ selects computations learned by the backbone rather than creating new algorithms. Together, these results show that the hidden state can control shared computation, with attention routing as a causal pathway.