Search papers, labs, and topics across Lattice.
This paper introduces autonugget, a Python package designed to provide a stable and efficient solution for ill-conditioned linear systems, which are common in machine learning applications. By leveraging Richardson extrapolation, autonugget combines multiple linear solves to enhance accuracy while maintaining compatibility with automatic differentiation frameworks like JAX. The key finding reveals that this approach significantly improves the numerical stability and performance of rapid prototyping algorithms compared to traditional single-nugget methods.
autonugget achieves superior accuracy in solving ill-conditioned linear systems by intelligently combining multiple linear solves, making it a game-changer for rapid algorithm prototyping.
Rapid prototyping of algorithms is a critical step in modern machine learning. Most algorithms exploit linear algebra, creating a need for lightweight numerical routines which -- while potentially sub-optimal for the task at hand -- can be rapidly implemented. For the numerical solution of ill-conditioned linear systems of equations, the standard solution for prototyping is Tikhonov-regularised inversion using a nugget. However, selection of the size of nugget is often difficult, and the use of data-adaptive procedures precludes automatic differentiation, introducing instabilities into end-to-end training. Further, while data-adaptive procedures perform multiple linear solves to select the size of nugget, only the result of one such solve is returned, which we argue is wasteful. This paper aims to circumvent the above difficulties, presenting autonugget; a Python package for automatic and stable numerical solution of linear systems suitable for rapid prototyping, and fully compatible with automatic differentiation using JAX. autonugget combines multiple linear solves using Richardson extrapolation to determine the solution of the ill-conditioned system, improving in accuracy over approximations based on a single nugget.
