Search papers, labs, and topics across Lattice.
This paper explores the use of the H-Score objective for feature extraction networks, addressing the challenges of estimating mutual information in low-data scenarios. By introducing unitary preconditioning with the fast Fourier transform (FFT), the authors demonstrate that selecting an appropriate basis rotation can significantly reduce truncation errors and enhance predictive performance. Experiments reveal that FFT preconditioning leads to up to a 50% reduction in normalized mean squared error across multiple datasets, highlighting its effectiveness in resource-constrained environments.
Unitary preconditioning with FFT can slash prediction errors by up to 50% in low-data regimes, transforming how we approach feature learning.
Mutual information (MI)-inspired feature learning techniques are capable of generating low-dimensional embeddings that retain nonlinear dependence structures, but direct estimations of MI suffer from noisy probability distribution estimates in the low-data regime. The H-Score objective, computed from second-order statistics, provides a practical proxy metric for training feature extraction networks. We prove that H-Score is invariant to invertible transformations in the unrestricted functional setting, but becomes sensitive to input basis rotations under constrained approximation classes. Consequently, we study unitary preconditioning for H-Score networks and show that selecting an appropriate basis rotation reduces finite-width truncation error by concentrating predictive dependence into fewer dominant modes. We identify the fast Fourier transform (FFT) as an effective data-independent, low-cost preconditioner for approximately stationary processes, where spectral structure induces concentration of the cross-covariance singular value spectrum. We introduce training-free metrics based on spectral entropy and cumulative dependence energy to quantify basis suitability and predict downstream inference gains prior to network training. Experiments across eight multivariate datasets demonstrate that FFT preconditioning is particularly useful in resource-constrained regimes, achieving up to 50% normalized mean squared error (NMSE) reduction, while the proposed metrics correlate with observed performance gains and correctly identify cases where spectral preconditioning is detrimental.
