Search papers, labs, and topics across Lattice.
This paper addresses the challenge of uniformly sampling matrices from the unit spectral norm ball, which is crucial for machine learning optimization. The authors prove that as matrix dimensions increase, all singular values converge to 1 almost surely, providing a theoretical foundation for their proposed sampling method. Experimental results validate both the convergence of singular values and the effectiveness of the proposed sampling techniques, particularly in the context of large language models.
All singular values of high-dimensional sampled matrices converge to 1, unlocking a new approach to sampling in machine learning optimization.
Motivated by an application in machine learning optimization, this paper focuses on the challenges of sampling a matrix uniformly from the unit spectral norm ball. It is proven that all singular values of sampled matrices converge to 1 almost surely as the matrix dimensions increase. This result provides the theoretical justification for a proposed simple sampling method applicable for large dimension sizes matching matrices found in modern large language models. Experimental results demonstrate both the convergence of the singular values, as well as the exact and proposed approximate sampling methods.
