Search papers, labs, and topics across Lattice.
The paper introduces LoopFormer, a looped Transformer architecture designed for budget-conditioned reasoning by training on variable-length trajectories. A shortcut-consistency training scheme is proposed to align trajectories of different lengths, ensuring shorter loops provide informative representations and longer loops refine them. Experiments on language modeling and reasoning benchmarks demonstrate LoopFormer's robust performance under compute constraints and graceful scaling with additional budget, suggesting looped Transformers are well-suited for adaptive language modeling.
Looped Transformers can adapt their computational depth on the fly, opening the door to budget-aware LLMs that trade compute for accuracy.
Looped Transformers have emerged as an efficient and powerful class of models for reasoning in the language domain. Recent studies show that these models achieve strong performance on algorithmic and reasoning tasks, suggesting that looped architectures possess an inductive bias toward latent reasoning. However, prior approaches fix the number of loop iterations during training and inference, leaving open the question of whether these models can flexibly adapt their computational depth under variable compute budgets. We introduce LoopFormer, a looped Transformer trained on variable-length trajectories to enable budget-conditioned reasoning. Our core contribution is a shortcut-consistency training scheme that aligns trajectories of different lengths, ensuring that shorter loops yield informative representations while longer loops continue to refine them. LoopFormer conditions each loop on the current time and step size, enabling representations to evolve consistently across trajectories of varying length rather than drifting or stagnating. Empirically, LoopFormer demonstrates robust performance on language modeling and reasoning benchmarks even under aggressive compute constraints, while scaling gracefully with additional budget. These results show that looped Transformers are inherently suited for adaptive language modeling, opening a path toward controllable and budget-aware large language models.
