Search papers, labs, and topics across Lattice.
This paper introduces the Gated Recurrent Transformer, a novel architecture that balances expressivity and memory efficiency by employing a recurrent depth mechanism that modulates updates through lightweight projections and gates. By iterating a shared core across multiple recurrences, the model achieves functional diversity without the need for a large number of unique layers, significantly reducing memory usage. Experimental results show that this approach matches or exceeds the performance of traditional transformer architectures while using fewer parameters and less memory, demonstrating a 10% increase in generation latency under certain constraints.
Achieving the accuracy of a 12-layer GPT-2 Small with just a 3-layer Gated Recurrent Transformer showcases a groundbreaking efficiency in model design.
Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.
