Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper reveals that the refresh interval alone is insufficient to define the refresh schedule for Kronecker-preconditioned optimizers, demonstrating that the timing of refreshes significantly impacts held-out loss even with fixed compute. Experiments with the Mousse optimizer on GPT-style models across various sizes show a learning-rate-dependent reversal in optimal refresh placement, with later refreshes favoring low learning rates and earlier ones favoring high learning rates. These findings underscore the necessity of reporting and comparing full refresh schedules across relevant learning rates to ensure robust optimization.
Refresh timing, not just frequency, dictates Kronecker-preconditioned LLM training performance, with optimal placement flipping based on learning rate.
Kronecker-preconditioned optimizers recompute a cached curvature eigendecomposition only periodically, and implementations and adaptive methods control this through the refresh frequency. We show that frequency leaves part of a schedule free: at a fixed number of refreshes, when they occur changes held-out loss at the same compute. Training GPT-style models with the Mousse optimizer across model sizes, we move the middle refreshes of a four-refresh schedule while fixing the first and last. On FineWeb the two placements exchanged rank as the learning rate rose (later placement was better at low rates, earlier placement at high ones), so a single-learning-rate comparison can report either ranking. The reversal held under controls including the published optimizer internals, and placement still mattered at the published refresh density. The learning rate at which the preference flips fell with model size, consistent with a held-out smaller model, and shifted with training length. A spectral-width staleness score predicted the same ranking at every learning rate and missed the reversal. Refresh schedules should be reported in full and compared across the learning rates at which they will be used.