Search papers, labs, and topics across Lattice.
This paper investigates the limitations of standard Transformers when pre-training on limited data, revealing that increasing model size beyond an optimal scale leads to overfitting and diminished generalization. By analyzing performance across various pre-training budgets and downstream tasks, the authors demonstrate that the optimal model size is highly dependent on both the data budget and the specific evaluation target. They introduce recursive Transformers, which effectively reuse shared blocks and utilize factorized embeddings, resulting in superior performance compared to standard Transformers under constrained data conditions.
Recursive Transformers can outperform standard models on limited data, revealing a new pathway to effective scaling in resource-constrained scenarios.
Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.
