Samuel Horváth

Interleaving expensive Muon spectral steps with cheap Lion sign descent not only cuts distributed training wall-clock time by 33%, but actually beats full Muon and AdamW on final perplexity.

Asynchronous pipeline parallelism can match synchronous training performance if the right optimizer is chosen, debunking myths about gradient delay instability.

Forget gradient projections – NESS sidesteps catastrophic forgetting by directly exploiting the null space of previous tasks, identified via small singular values, to constrain weight updates.

LoRA fine-tuning just got a memory-efficient upgrade: LoRSum matches or beats standard LoRA performance by reformulating optimization as a proximal problem and using diagonal K-FAC approximations, all without expensive SVD.