Search papers, labs, and topics across Lattice.
The paper introduces Loopie, a groundbreaking looped Transformer architecture that includes two Mixture-of-Experts (MoE) models, significantly enhancing performance while maintaining a manageable parameter count. By addressing the long-standing challenge of efficiency in looped Transformers, Loopie demonstrates that it can outperform traditional models trained with equivalent compute resources, as evidenced by extensive ablation studies. Notably, Loopie achieves gold-medal performance in reasoning tasks at prestigious competitions, showcasing its advanced capabilities in real-world applications.
Loopie outperforms traditional Transformers by leveraging a novel architecture that maximizes efficiency without sacrificing reasoning power, achieving gold-medal performance in competitive settings.
We present Loopie, the most powerful looped Transformer to date. The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6Bparameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. Our novel post-training pipeline equips Loopie with strong reasoning abilities. At the 2025 IMO and IPhO, Loopie achieves gold-medal performance without tools.
