Search papers, labs, and topics across Lattice.
This paper introduces the Puro-2B model collection, which is trained from scratch on consumer-grade RTX 5090 GPUs using an innovative open pretraining recipe that significantly reduces costs to under $6.9K while achieving performance close to Qwen2.5-1.5B. The authors establish a Puro Cost Scaling Law that correlates training expenses with model performance, indicating that a budget of approximately $4.4K is sufficient to match Qwen2-1.5B performance. Additionally, they explore the impact of pretraining data curricula on downstream performance, providing a comprehensive view of the training process beyond just model weights.
Achieving near state-of-the-art performance for under $7K opens the door for cost-effective language model training accessible to the broader research community.
Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.
Tsinghua AI