Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
To bridge the convergence gap between backpropagation and memory-efficient zero-order optimization (ZOO), the authors scale perturbation counts via 1SPSA and introduce 1.5-SPSA, which adds a single unperturbed forward pass per step to compute a lightweight diagonal preconditioner in probe-space. This setup circumvents the need for stored activations, backward passes, and optimizer states while damping unstable, high-curvature parameter directions. Across OPT and Qwen3 benchmarks, 1.5-SPSA matches or exceeds backpropagation performance, notably achieving +3.1% higher accuracy than MeZO on OPT-13B in just 70 steps compared to MeZO's 100,000 steps while training up to OPT-30B on commodity A100s.
Probe-space preconditioning compresses zero-order LLM optimization from 100,000 steps down to 70, outperforming standard backpropagation on OPT-13B while slashing training memory requirements by 90%.
Backpropagation (BP) dominates deep learning but imposes a massive memory tax. For example, training OPT-30B with Adam requires $\approx$ 600GB of GPU memory (assuming batch size 8 and sequence length 2048). Alternatively, zero-order optimization (ZOO) trains in inference-mode (requiring only $\approx$ 60GB for the same model): no stored activations, no gradients, and no optimizer states. However, ZOO convergence has lagged behind BP. In this work, we evaluate two methods to close this gap. First, we show that reallocating training compute budget from many steps to large effective batch sizes with many perturbations (or probes) but fewer steps, allows 1SPSA (Spall, 1992) to outperform zero order methods like MeZO (Malladi et al., 2023) with less training compute. Next, we introduce 1.5-SPSA, adding a single "clean" forward-pass per step to 1SPSA to calculate a cheap diagonal preconditioner in probe-space, which improves convergence rate and convergence by down-weighting high curvature directions. Benchmarking on 6 post-training datasets on both Qwen3 and OPT model families, we show that 1.5-SPSA achieves State-of-the-Art results over previous ZOO solvers with much less optimization steps. For example, we train OPT-13B (for direct comparison to MeZO) and find 1.5-SPSA achieves +3.1% accuracy on SST-2 over both MeZO and BP in only 70 steps vs. MeZO's 100,000 steps. Finally, we combine an 8-bit-packing random generator, triton fused unpack/apply kernels, and distributed parallelism to achieve fast and stable training of models as large as OPT-30B in-place on commodity GPUs (e.g. A100).
Stanford HAI