Search papers, labs, and topics across Lattice.
This paper introduces Spheroid BlockTrain, a decentralized training protocol that partitions AI models into independently trainable blocks, allowing for local optimization while still aligning with a global target. The method demonstrates competitive performance on byte-level WikiText, achieving a cross-entropy of 1.359, which is only slightly behind a traditional end-to-end Transformer model. Additionally, BlockTrain showcases significant improvements in inference efficiency, outperforming standard TCP pipelines by emitting full sequences per traversal instead of individual tokens.
Decentralized training can rival centralized models, achieving competitive performance while democratizing access to AI development resources.
Frontier AI training is increasingly shaped by access to dense, centrally controlled accelerator clusters. This creates a structural advantage for hyperscalers and large centralized laboratories, and makes open or independent AI efforts depend on scarce capital, privileged infrastructure, and data-center geography. We present Spheroid BlockTrain, a decentralized training protocol in which a model is partitioned into independently trainable blocks, each optimized on a local objective derived from the same global target and composed at inference into one model. On byte-level WikiText, BlockTrain reaches cross entropy 1.359 (perplexity 3.89), within about 0.04 CE of a same-setup end-to-end Transformer reference, while each active worker trains only one block and avoids full-model optimizer state. A shared six-worker block training run reaches CE 1.385 by averaging same-block updates into one assembled model. HTTP/TCP transport experiments move real serialized checkpoints and updates, including a public-IP three-host run that improves CE from 5.580 to 1.811 while moving 15.22 GB. For inference, the current BlockTrain path uses one block-stack traversal per full output and serves over direct TCP across three public-network GPU hosts up to a 75.80B-parameter logical fp16 shape, outperforming a matched plain-autoregressive TCP pipeline baseline because it emits a full sequence per WAN pipeline traversal rather than one token per traversal.
