Search papers, labs, and topics across Lattice.
To probe parameter-compute trade-offs under severe data constraints, the authors evaluate Looped GPT-BERT, which integrates hybrid masked next-token and causal language modeling objectives with depth-wise recurrent weight tying on the 7.48M-token BabyLM 2026 Strict-small track. Their 12.18M-parameter architecture鈥攅xecuting 12 recurrent traversals across just four physical layers鈥攎atches standard GPT-2 and non-looped GPT-BERT baselines on BLiMP and GLUE benchmarks. Crucially, ablation analyses indicate that while recurrent computation substantially aids syntactic acquisition, restricting physical depth ultimately imposes a hard bottleneck on general representational capacity.
Recycling just four transformer layers through twelve recurrent iterations matches full-depth baselines on core linguistic benchmarks, exposing precisely where compute can鈥攁nd cannot鈥攕ubstitute for raw parameter capacity.
When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliver comparable performance. We study Looped GPT-BERT in the BabyLM 2026 Strict-small setting, combining GPT-BERT's masked next-token and causal language-modeling objectives with depth-wise parameter sharing. We train on a preprocessed 7.48M-word English corpus and compare objective ratios, non-looped and looped architectures, and loop counts. Our final $4\times12$ model uses four physical layers for twelve recurrent traversals and contains 12.18M parameters. The BabyLM 2026 leaderboard reports an Overall Average of 35.42 and an NLP Average of 48.48. Compared with public BabyLM 10M Strict-small GPT-2 and GPT-BERT baselines, it achieves comparable performance on selected linguistic and downstream metrics, including BLiMP and GLUE, with fewer parameters. The loop ablations show that additional recurrent computation can improve training and preserve strong performance on selected linguistic tasks, whereas poorer performance on other tasks may reveal an inherent limitation of the looped design: using only a few physical layers restricts the model's representational space.
