Search papers, labs, and topics across Lattice.
This paper introduces the Gated Recurrent Transformer (GRT), a novel architecture that balances expressivity and memory efficiency by employing a recurrent depth mechanism with fixed-depth prelude and coda blocks. By utilizing a lightweight projection and an elementwise update gate, the GRT allows for functional specialization across fewer layers, achieving comparable accuracy to a 12-layer GPT-2 Small model while significantly reducing memory usage and parameter count. The findings reveal that GRT can outperform traditional transformer architectures in terms of quality and efficiency, demonstrating a viable path for optimizing large-scale language models.
Gated Recurrent Transformers achieve 63% fewer parameters and 59% less peak memory while matching the accuracy of much larger models, redefining efficiency in language modeling.
Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.
