Search papers, labs, and topics across Lattice.
This paper critiques the widespread use of the Gaussian kernel in Gaussian process regression, highlighting its significant drawbacks. The authors demonstrate that the Gaussian kernel leads to unrealistically small conditional variances, resulting in catastrophic overconfidence in predictive uncertainty, and is prone to numerical ill-conditioning. They argue that the inherent smoothness of analytic kernels, including the Gaussian, is problematic and suggest that such kernels should be avoided in favor of alternatives that better capture the complexities of real-world data.
The Gaussian kernel's smoothness can lead to catastrophic overconfidence in predictive uncertainty, making it a risky default choice for regression tasks.
Kernels measure similarity or correlation in tasks such as regression and classification. The Gaussian kernel, other names of which include squared exponential and radial basis function kernel, is one of the most popular in Gaussian process regression. We argue that the Gaussian kernel is best avoided and should never be used as a default. The argument rests on two results demonstrating that the Gaussian kernel is extremely brittle. First, the Gaussian kernel gives rise to a conditional variance that is unrealistically small. If the variance is used to quantify predictive uncertainty, catastrophic overconfidence is almost inevitable. Second, a small variance goes hand in hand with numerical ill-conditioning, so that to use the Gaussian kernel in practice requires tricks such as nugget terms that effectively modify the underlying regression or classification model. These problems are caused by the unnatural smoothness of the Gaussian kernel, a fact we are far from the first to take notice of. The problem is not the Gaussian form itself but the analyticity of the kernel: Our argument is more broadly that analytic kernels are best avoided. For stationary kernels analyticity is essentially equivalent to an exponential decay of the spectral density.
