Search papers, labs, and topics across Lattice.
This paper investigates the smallest eigenvalue of the ReLU neural tangent kernel (NTK) Gram matrix, deriving a dimension-free lower bound that scales with the projective separation of unit vectors. The authors establish that the smallest eigenvalue is tightly bounded by the expression \( \lambda_{\min}(H) = \Theta(\Delta_\pm/\sqrt{\log n}) \), where \( \Delta_\pm \) represents the minimum distance between the vectors in either direction. These findings not only provide a fundamental understanding of the NTK's behavior but also highlight the conditions under which these bounds hold, offering insights into the geometry of neural networks.
The smallest eigenvalue of ReLU NTK Gram matrices is tightly bounded, revealing critical geometric insights into neural network behavior.
For $n$ unit vectors $x_1,\ldots,x_n \in \mathbb{R}^d$, we study the continuous ReLU derivative Gram matrix $H$, whose entries are obtained by averaging pairwise gated inner products over a standard Gaussian direction. Writing $ \Delta_\pm := \min_{i \neq j} \min\{ \|x_i-x_j\|_2, \|x_i+x_j\|_2 \} $ for their projective separation, we prove the universal dimension-free lower bound $ \lambda_{\min}(H) = \Omega( \Delta_\pm/\sqrt{\log n} ) $. Conversely, we construct worst-case families satisfying the matching upper bound $ \lambda_{\min}(H) = O( \Delta_\pm/\sqrt{\log n} ) $, showing that this rate is tight up to universal constants.
