Search papers, labs, and topics across Lattice.
This paper addresses the issue of representation collapse in Self-Supervised Learning (SSL) by introducing a family of statistical regularizers that enforce a uniform distribution on the unit hypersphere without the stochasticity of existing methods. By analytically integrating out random projections, the authors derive deterministic objectives for Maximum Mean Discrepancy (MMD), Kernel Stein Discrepancy (KSD), and Kullback-Leibler (KL) divergence, which enhance optimization stability and convergence speed. Empirical results demonstrate that these full-dimensional objectives outperform traditional stochastic regularizers on benchmark datasets like ImageNet and Galaxy10, while also influencing the latent space geometry for improved task performance.
Removing stochastic noise from statistical regularizers leads to faster convergence and superior performance in Self-Supervised Learning tasks.
In Self-Supervised Learning (SSL), preventing representation collapse by explicitly enforcing a uniform distribution on the unit hypersphere has proven to be effective. However, current frameworks typically rely on sliced statistical regularizers such as SIGReg (used in LeJEPA) and SUSReg (used in SPHERE-JEPA), which approximate this continuous objective via Monte Carlo sampling along random 1D directions. This stochasticity injects projection variance into the training gradients, destabilizing optimization, and hindering convergence. In this work, we first show that analytically integrating out these random projections natively yields a deterministic Maximum Mean Discrepancy (MMD), bypassing the variance of sliced methods. Motivated by this equivalence, we formulate full-dimensional objectives for MMD, Kernel Stein Discrepancy (KSD), and Kullback-Leibler (KL) divergence directly on the sphere to enforce a uniform distribution. To prevent spatial bias, we equip these tests with rotationally invariant kernels constructed via spectral theory, systematically evaluating two canonical families: smooth exponential decay (Heat) and strict frequency cutoff (Bandlimited) filters. Empirically, removing projection-induced noise results in more stable optimization, faster convergence, and consistent improvements over stochastic sliced regularizers on ImageNet and Galaxy10. Furthermore, we reveal that the choice of the statistical test shapes the geometry of the learned latent space: MMD and KSD favor locally clustered organization suitable for object-centric domains, whereas the continuous KDE-based KL divergence promotes fine-grained instance separation, yielding the strongest results on unclustered procedural texture retrieval.
