Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
SoftServe introduces a novel family of Quasi-Newton methods designed for deep learning by deriving positive-definite curvature estimates from a variational objective, effectively handling non-convexity and negative curvature without line searches. Diagonal and Kronecker-factored variants are developed to ensure positive definiteness by construction and scale to massive neural networks. This approach, utilizing stable coupled Newton-Schulz iterations for matrix operations, achieves superior performance on ill-conditioned problems compared to established optimizers.
Quasi-Newton methods now conquer deep learning's non-convexity and scale, outperforming Adam and others on challenging tasks like 136M-parameter physics-informed models.
Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.