Search papers, labs, and topics across Lattice.
Across an end-to-end 30B parameter mixture-of-experts (MoE) training pipeline, this work investigates whether optimal pretraining checkpoints鈥攋udged by loss and benchmark scores鈥攃onsistently yield the best models after downstream supervised fine-tuning (SFT). The authors find that standard pretraining metrics fail to predict downstream performance, as the highest-scoring pretraining checkpoints often underperform post-SFT. Instead, downstream training success is strongly predicted by solution density, where checkpoints occupying flatter loss basins that tolerate local weight perturbations transfer significantly better across the full alignment stack.
Pretraining loss is a deceptive selection metric: at 30B MoE scale, downstream SFT performance is governed not by benchmark scores, but by the checkpoint's solution density under local weight perturbations.
Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.
