Search papers, labs, and topics across Lattice.
This paper introduces Mixture of Training (MoT), a novel approach that decomposes language model pre-training into smaller, independently trainable jobs, which can be recombined into a larger model. The authors demonstrate that by training contiguous layer blocks within a frozen pretrained aligner scaffold, they can achieve quality parity with a monolithic baseline while processing more aggregate tokens and reducing the critical path for training. The findings suggest that MoT offers a viable framework for exploring reusable training units, although it is not positioned as a complete replacement for traditional pre-training methods.
Independently trained depth slices can be recombined to match the performance of monolithic models, revealing a new avenue for efficient language model training.
We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass. On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slices can be recomposed into a usable language model, and a quality-parity schedule reaches the same reported perplexity as the monolithic baseline. This parity setting processes more aggregate tokens and has a shorter idealized layer-equivalent critical path after aligner preparation; its effective compute advantage depends on reusing the aligner across runs. We therefore present MoT not as a general replacement for monolithic pre-training, but as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.
