Search papers, labs, and topics across Lattice.
This paper analyzes the generalization properties of the Lion optimizer using algorithmic stability, proving a generalization error bound of $O(\frac{1}{N\tau^T})$. It also surprisingly shows that SignSGD has the same generalization error. To improve generalization, the authors introduce Cautious Lion (CLion), which cautiously uses the sign function, achieving a lower generalization error of $O(\frac{1}{N})$ and a convergence rate of $O(\frac{\sqrt{d}}{T^{1/4}})$ for nonconvex stochastic optimization.
Lion optimizer's generalization error is worse than you thought ($O(\frac{1}{N\tau^T})$), but a simple tweak (CLion) can fix it, achieving $O(\frac{1}{N})$ with faster convergence.
Lion optimizer is a popular learning-based optimization algorithm in machine learning, which shows impressive performance in training many deep learning models. Although convergence property of the Lion optimizer has been studied, its generalization analysis is still missing. To fill this gap, we study generalization property of the Lion via algorithmic stability based on the mathematical induction. Specifically, we prove that the Lion has a generalization error of $O(\frac{1}{N\tau^T})$, where $N$ is training sample size, and $\tau>0$ denotes the smallest absolute value of non-zero element in gradient estimator, and $T$ is the total iteration number. In addition, we obtain an interesting byproduct that the SignSGD algorithm has the same generalization error as the Lion. To enhance generalization of the Lion, we design a novel efficient Cautious Lion (i.e., CLion) optimizer by cautiously using sign function. Moreover, we prove that our CLion has a lower generalization error of $O(\frac{1}{N})$ than $O(\frac{1}{N\tau^T})$ of the Lion, since the parameter $\tau$ generally is very small. Meanwhile, we study convergence property of our CLion optimizer, and prove that our CLion has a fast convergence rate of $O(\frac{\sqrt{d}}{T^{1/4}})$ under $\ell_1$-norm of gradient for nonconvex stochastic optimization, where $d$ denotes the model dimension. Extensive numerical experiments demonstrate effectiveness of our CLion optimizer.
