Search papers, labs, and topics across Lattice.
This paper introduces Kernel-Thinned Mean Field Langevin Dynamics (KT-MFLD), a novel algorithm for minimizing entropy-regularized objectives over probability distributions. KT-MFLD reduces the computational complexity of standard MFLD from O(N^2) to O(N^(3/2)) by having each particle interact only with a thinned particle coreset. Theoretical analysis demonstrates that KT-MFLD achieves similar convergence guarantees to MFLD, which is further validated empirically on tasks like training student-teacher neural networks and Bayesian posterior computation.
Get faster convergence in entropy-regularized learning tasks: Kernel Thinning slashes the computational cost of Mean Field Langevin Dynamics while preserving its theoretical guarantees.
Several important learning tasks can be formulated as minimizing an entropy-regularized objective over an appropriate space of probability distributions. Mean-field Langevin dynamics (MFLD) facilitate computation in this general context, casting the minimizer as the invariant distribution of a McKean--Vlasov process, which can be numerically discretized using $N$ particles and thus simulated. However, simulating this interacting particle system has computational complexity of order $N^2$. Motivated by recent research into \emph{kernel thinning}, we propose \texttt{KT-MFLD}, in which each particle interacts only with a thinned particle coreset of size $\mathcal{O}(N^{\frac{1}{2}})$. \texttt{KT-MFLD} thus reduces the computational complexity to order $N^{\frac{3}{2}}$ while, under mild regularity conditions, achieving the same convergence guarantees (up to logarithmic factors) as MFLD. Our theoretical analysis is empirically confirmed on tasks including the training of student-teacher neural networks, quantization with maximum mean discrepancy, and computation of predictively-oriented posteriors in a post-Bayesian framework.
Microsoft Research