Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the use of Gaussian averaging as a smooth surrogate for quantized neural networks, establishing a local dimension-dependent bound on the difference between quantized and smoothed models under bounded local oscillation. The authors compute closed-form Gaussian averages for ReLU and sign activation functions, demonstrating how this approach can enhance stability in high-dimensional binary perceptrons through layer-preactivation aggregation. Key results show that this method effectively links Gaussian smoothing to the stability analysis of discontinuous networks, improving both inference and training processes.
Gaussian smoothing can significantly enhance the stability of quantized neural networks, revealing a surprising connection between local oscillation and model performance.
We study Gaussian averaging as a smooth surrogate for quantized neural models. Under bounded local oscillation, we derive a local dimension-dependent bound on |f-g|, linking Gaussian smoothing to the stability analysis of discontinuous networks. We compute closed-form Gaussian averages of the rectified linear unit (ReLU) and sign activation functions, and illustrate the mechanism on a high-dimensional binary perceptron, where layer-preactivation aggregation under an explicit quantization-noise surrogate yields the Gaussian envelope used in inference-side smoothing and training-side smooth surrogate gradients.