Kristi Topollai

Forget hyperparameter tuning: periodically restarting outer momentum in distributed training can widen the stable range of learning rates and momentum, making training more robust.

Quantizing optimizer states in LLM pre-training introduces "staleness," but strategically timed resets can recover lost performance and reduce memory footprint.