Search papers, labs, and topics across Lattice.
This paper investigates the scaling behavior of sketched linear models in contrastive representation learning, particularly focusing on a paired Gaussian latent-variable framework. By deriving a risk decomposition and an explicit scaling law that incorporates sketch dimension, sample size, and effective optimization horizon, the authors reveal how the interaction between two views alters the scaling dynamics compared to standard linear regression. The findings not only enhance theoretical understanding but also offer practical guidance for optimizing model size, data, and compute resources in contrastive learning scenarios.
Scaling laws for contrastive learning reveal that learning interactions between views fundamentally alters optimization dynamics compared to linear regression.
Scaling laws describe how learning performance varies with model size, data size, and compute. While recent theoretical work has established scaling laws for sketched linear regression, much less is understood for contrastive representation learning. In this paper, we study a sketched linear model for contrastive learning under a paired Gaussian latent-variable setup. The learner observes only sketched views of two correlated variables and trains a bilinear contrastive score by full-batch empirical gradient descent. We analyze a Gaussian-negative quadratic contrastive surrogate under aligned power-law spectra and a contrastive source condition, where we derive a risk decomposition into irreducible risk, approximation error, GD bias, GD variance, and a cross term. The cross term is controlled by the bias and variance and therefore does not affect the upper-bound scaling. Our main theorem gives an explicit scaling law with respect to sketch dimension $M$, sample size $N$, and effective optimization horizon $L_{\mathrm{eff}}纬$. Compared with standard linear-regression scaling laws, the contrastive setting must learn interactions between two views, and this changes how optimization and finite-sample noise scale with model size, data, and training time. This provides a first theoretical step toward understanding scaling behavior in contrastive learning and gives guidance for balancing model size, data, and optimization compute.
