Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates whether hierarchically structured, shared-weight recurrence in Transformers (HRM-LM) can achieve comparable performance to standard independent-layer stacking. They replace L independent Transformer layers with a two-speed recurrent pair (Fast and Slow modules) unrolled for M steps with shared parameters. The key finding is a significant performance gap favoring independent-layer Transformers over the parameter-matched HRM-LM, even after multiple independent runs.
Hierarchical recurrence with shared weights can't match the representational power of standard independent-layer Transformers, even when parameter counts are matched.
We present an empirical study of whether hierarchically structured, shared-weight recurrence can match the representational quality of independent-layer stacking in a Transformer-based language model. HRM-LM replaces L independent Transformer layers with a two-speed recurrent pair: a Fast module operating at every step for local refinement, and a Slow module operating every T steps for global compression. This recurrent hierarchy is unrolled for M = N x T steps with shared parameters. The central and most robust finding, supported by a parameter-matched Universal Transformer ablation (UniTF, 1.2B) across five independent runs, is a sharp empirical gap between the two approaches.