Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the computational bottleneck of Cholesky decompositions in Sparse Variational Gaussian Processes (SVGPs) by developing an inverse-free SVGP formulation that relies solely on matrix multiplications. They introduce a better-conditioned variational bound and derive a natural-gradient update for an auxiliary parameter, enabling stable and faster optimization using only matrix multiplications. Experiments on regression and classification tasks show their method achieves comparable performance to traditional SVGPs and can be faster with proper tuning, making it a viable drop-in replacement.
Ditch the Cholesky decomposition: this work makes Sparse Variational Gaussian Processes (SVGPs) matrix-multiply friendly, unlocking faster and more scalable GP training on modern hardware.
Gaussian processes (GPs) offer appealing properties but are costly to train at scale. Sparse variational GP (SVGP) approximations reduce cost yet still rely on Cholesky decompositions of kernel matrices, ill-suited to low-precision, massively parallel hardware. While one can construct valid variational bounds that rely only on matrix multiplications (matmuls) via an auxiliary matrix parameter, optimising them with off-the-shelf first-order methods is challenging. We make the inverse-free approach practical by proposing a better-conditioned bound and deriving a matmul-only natural-gradient update for the auxiliary parameter, markedly improving stability and convergence. We further provide simple heuristics, such as step-size schedules and stopping criteria, that make the overall optimisation routine fit seamlessly into existing workflows. Across regression and classification benchmarks, we demonstrate that our method 1) serves as a drop-in replacement in SVGP-based models (e.g., deep GPs), 2) recovers similar performance to traditional methods, and 3) can be faster than baselines when well tuned.