Search papers, labs, and topics across Lattice.
Agora introduces a novel system for training large language models using a decentralized approach that leverages heterogeneous, consumer-grade GPUs connected via the internet. By employing bandwidth-efficient pipeline-parallel model sharding and multi-party collective operations, Agora enables the collaborative training of models without any single entity holding the complete model weights. The first implementation, Pluralis-8B, successfully trained an 8.6B-parameter model on 500B tokens with efficiency comparable to centralized systems, demonstrating the feasibility of open-source frontier training.
Collective training of large language models can now be achieved with consumer GPUs, making frontier AI development accessible to a broader community.
Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homogeneous accelerators, high-speed interconnects, and a single orchestrating entity. Frontier model development is thereby concentrated among the few groups able to assemble such clusters. Meanwhile, an enormous pool of compute remains unusable for training: consumer and professional GPUs that are heterogeneous, preemptible, individually owned, and connected only by the internet. We present Agora, a system that makes efficient use of this compute. Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations. Each participant holds only one stage of the model, and no single party ever possesses the full weights. We term this setup Protocol Learning: it enables collectively trained, collectively owned models, opening a path to open-source frontier training with economic sustainability. This report presents the outcome of a research effort spanning communication-efficient parallelism, asynchronous optimization, and fault-tolerant systems design. It culminates in the first demonstration of its kind: Pluralis-8B, an open, permissionless pretraining run of an 8.6B-parameter model on 500B tokens of FineWeb-Edu. The model was trained over 40 days by 330 contributor nodes, predominantly consumer GPUs on internet connections, joining and leaving throughout. The run sustained ~170k tokens/s and 4.2 tokens per TFLOP of pooled compute, 63% of the efficiency of a centralized H100 baseline, and converged to within a small margin of a centralized reference run.
Amazon Science