Search papers, labs, and topics across Lattice.
This paper introduces SOAP-Bubbles, a novel approach to structured weight uncertainty in neural networks that leverages the Eigenspace-VON (EVON) optimizer. By adapting the existing IVON method within the eigenspace of the SOAP preconditioner, the authors achieve a non-diagonal covariance estimation without incurring significant additional costs or requiring major changes to existing training pipelines. The results demonstrate that EVON not only recovers the exact Gaussian covariance for logistic regression but also outperforms traditional diagonal-covariance methods in language model pretraining tasks.
EVON transforms structured weight uncertainty into a practical tool, yielding superior performance in language model pretraining without the overhead of complex implementations.
Structured weight-uncertainty can improve many aspects of deep learning, but it remains costly to estimate and difficult to implement. Here, we show that these issues can be addressed by adapting the SOAP optimizer. Our key idea is to run IVON, an existing diagonal-covariance variational method, in the eigenspace of SOAP's preconditioner and then use the preconditioner to transform the diagonal estimate into a non-diagonal covariance. The resulting method has costs similar to those of SOAP and requires no drastic changes to training pipelines. We call the posteriors obtained in this way SOAP-Bubbles and our new optimizer Eigenspace-VON (EVON). We show that, for logistic regression, EVON recovers the exact Gaussian covariance and that, for language model pretraining, it yields significantly better results than existing diagonal-covariance methods. Our work makes it easier to estimate more expressive posterior distributions for deep learning at scale.
