Search papers, labs, and topics across Lattice.
Covenant-72B, a 72B parameter LLM, was pre-trained in a globally distributed, permissionless setting using the SparseLoCo optimizer and a live blockchain protocol to manage dynamic peer participation. This work demonstrates the feasibility of large-scale, democratized pre-training by allowing open participation without whitelisting. The resulting model, trained on 1.1T tokens, achieves performance competitive with centralized models trained on similar or greater compute.
Democratized LLM pre-training is now a reality: Covenant-72B proves you can train a competitive 72B model with untrusted peers over the internet, opening the door to broader participation and reduced costs.
Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run.
