Search papers, labs, and topics across Lattice.
This paper revisits SignSGD, addressing its generalization gap by analyzing it through a 1-bit quantization and dithering lens. They derive a small-batch convergence rate under unimodal symmetric gradient noise, introduce annealed Gaussian noise as a dithering mechanism, and adapt the SWATS strategy for learning rate calibration. Experiments on ResNet-18 demonstrate that pre-sign dithering outperforms Adam on CIFAR-100, and a calibrated switch between SignSGD and SGD achieves state-of-the-art accuracy on CIFAR-10 compared to pure SGD and SignSGD.
SignSGD can beat Adam and even SGD with a few simple tweaks, proving that 1-bit quantization doesn't have to mean sacrificing accuracy.
SignSGD compresses each stochastic gradient coordinate to a single bit, offering substantial memory and communication savings, but its 1-bit quantization removes magnitude information and is known to leave a generalization gap relative to well-tuned SGD. We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements. First, we derive a small-batch convergence rate for SignSGD under unimodal symmetric gradient noise using a signal-to-noise weighted stationarity measure, removing the large-batch assumption of prior analyses. Second, we inject annealed Gaussian noise before the sign operator, which acts as a classical dithering mechanism and probabilistically restores magnitude information lost to hard thresholding. Third, we adapt the SWATS strategy to sign-based updates with a projection-based learning-rate calibration that smoothly transitions from SignSGD to SGD. Single-worker experiments on ResNet-18 isolate optimizer effects from communication aspects: pre-sign dithering surpasses Adam on CIFAR-100, and the calibrated switch reaches 92.18% test accuracy on CIFAR-10, outperforming both pure SGD 91.38% and pure SignSGD with momentum 90.82%.
