Search papers, labs, and topics across Lattice.
This paper investigates the benefits of adaptive per-layer looping and gated memory banks in transformers, comparing their impact on mathematical and commonsense reasoning. Looping is found to primarily improve mathematical reasoning, while memory banks are more effective for commonsense tasks, relative to FLOP-matched baselines. The combination of both mechanisms leads to performance gains over an iso-FLOP baseline with more layers, and analysis reveals layer specialization in utilizing looping and memory.
Looping helps transformers think harder on math problems, while memory lets them remember more commonsense facts, and combining both beats simply scaling up layers.
Chain-of-thought (CoT) prompting enables reasoning in language models but requires explicit verbalization of intermediate steps. Looped transformers offer an alternative by iteratively refining representations within hidden states. This parameter efficiency comes at a cost, as looped models lack the storage capacity of deeper models which use unique weights per layer. In this work, we investigate transformer models that feature both adaptive per-layer looping, where each transformer block learns to iterate its hidden state via a learned halting mechanism, and gated memory banks, that provide additional learned storage. We find that looping primarily benefits mathematical reasoning, while memory banks help recover performance on commonsense tasks compared to parameter and FLOP matched models. Combining both mechanisms yields a model that outperforms an iso-FLOP baseline, with three times the number of layers, across math benchmarks. Analysis of model internals reveals layer specialization: early layers learn to loop minimally and access memory sparingly, while later layers do both more heavily.
