Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a unified software tool that leverages CESTAC to analyze the numerical stability of deep learning operators, addressing the critical issue of numerical approximation errors in finite-precision arithmetic. The tool allows for numerical validation in a single computation pass while identifying sources of instability and monitoring stability throughout training and inference. Validation across various tasks demonstrates its effectiveness in detecting polluted operators, highlighting its potential to enhance the reliability of deep learning systems.
Numerical instability in deep learning could be effectively monitored and mitigated with a single pass using a new software tool that identifies and addresses errors in real-time.
Finite-precision arithmetic unavoidably introduces numerical approximation errors. Numerical computations may use insufficient precision or an improper formulation, which leads to numerical instability. In this paper, we introduce the first unified software tool that integrates CESTAC for detecting the numerical stability of deep learning operators. Our developed software not only enables numerical validation with a single computation pass but also detects the sources of numerical instability and provides numerical stability monitoring during deep learning training and inference. We verified its effectiveness on the detection of polluted operators with injected numerical instabilities across various tasks. We believe that our developed method and tool provide valuable insights into developing numerically stable computing kernels, which are particularly critical for numerically stable and efficient deep learning training and inference.