Search papers, labs, and topics across Lattice.
This paper investigates the optimization dynamics of Stochastic Gradient Descent (SGD) applied to Score-based Generative Models (SGMs), revealing a non-convex convergence rate that explicitly incorporates schedule-dependent weighting factors. Additionally, it employs a Neural Tangent Kernel analysis for overparameterized two-layer ReLU networks, deriving score-approximation error bounds along the SGD trajectory. The findings highlight the critical influence of reweighting factors on score approximation error, offering theoretical insights that can inform practical weighting strategies in SGM training.
Non-convex convergence rates for SGD in score-based generative models reveal how reweighting choices critically impact training efficiency and output quality.
Score-based Generative Models (SGMs) have achieved impressive performance in data generation across a wide range of applications. While the statistical properties of their sampling procedures are increasingly well understood, the optimization dynamics underlying their training remain less explored. SGMs are typically trained by minimizing a weighted denoising scorematching objective, yet optimization guarantees with stochastic gradients remain limited. In this work, we study Stochastic Gradient Descent (SGD) for SGMs, contributing results in two complementary regimes. First, for general score parameterizations, we establish a non-convex convergence rate for SGD on the weighted denoising score-matching objective, with explicit dependence on the schedule-dependent weighting factors. Second, for overparameterized two-layer ReLU networks, we develop a Neural Tangent Kernel analysis tailored to diffusion training with stochastic gradients, yielding score-approximation error bounds along the SGD trajectory. Finally, our analysis quantifies the role of the reweighting factor in the score approximation error, providing theoretical guidance for weighting choices used in practice.
