Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper systematically disentangles topology, input injection, and recurrent-state design in looped Transformer models, training from scratch with matched scales to isolate their effects. The prelude-recurrent block-coda sandwich architecture and specific input injection strategies consistently improve performance on context extraction and reasoning, while other state designs yield mixed results. The Huginn-style architecture, incorporating these principles, significantly outperforms the Ouro baseline and transfers its advantages to Mixture-of-Experts models, achieving competitive performance with substantially fewer resident parameters.
A specific looped Transformer architecture and input injection strategy unlock significant gains in reasoning and context utilization, outperforming larger models with drastically fewer parameters.
Looped language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization. We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test whether the resulting design principles transfer to resident- and active-parameter-matched Mixture-of-Experts (MoE) models. The prelude-recurrent block-coda sandwich architecture consistently improves performance on context extraction and reasoning-intensive tasks. Input injection primarily enhances knowledge retrieval and context utilization, whereas random initialization and separate H/L recurrent states provide mixed or negative returns. Taken together, these results favor the Huginn-style design: at the 730M-parameter and 336B-token scale, the complete Huginn architecture outperforms Ouro across all ten benchmarks. The advantages of Huginn over Ouro transfer to MoE models. At the 8BA0.8B (32BA3B when unrolled) scale trained on 500B tokens, Huginn outperforms Ouro on nearly all benchmarks spanning knowledge, reasoning, and coding, while also exhibiting more balanced expert routing. Under matched training and inference FLOPs, the Huginn-style model uses 75% fewer resident parameters than a 112-layer feedforward MoE baseline. Despite its smaller physical parameters, it outperforms the feedforward baseline on DROP and GSM8K, matches it on MATH500, and remains competitive on the other benchmarks.