Search papers, labs, and topics across Lattice.
This paper identifies a significant client-independent second-order stationary bias component in stochastic SCAFFOLD, demonstrating that while client averaging mitigates the leading $O(\gamma/N)$ bias, it fails to eliminate the persistent $O(\gamma^2)$ bias under certain conditions. Through rigorous analysis, the authors derive the coefficient of this bias, revealing its dependence on the curvature of the objective function and the behavior of gradient noise. Numerical experiments validate the theoretical findings, highlighting the implications of this bias as client counts increase in a full-participation setting with fixed local-step counts.
Client averaging may suppress leading biases, but a hidden second-order bias persists, challenging assumptions in distributed optimization.
Existing constant-step analysis of stochastic \Scaf{} identifies a leading $O(\gamma/N)$ stationary mean bias and shows that higher-order bias can persist as the client count increases, but does not identify the first client-independent contribution at coefficient level. For full-participation stochastic \Scaf{} with one-dimensional homogeneous clients, fixed local-step count $H$, and bounded additive gradient noise, we prove, uniformly over $N\ge2$, $$ \begin{aligned} \mathbb{E}_{\pi_{\gamma,N,H}}[x]-x^\star ={}&-\frac{f'''(x^\star)\sigma^2}{4f''(x^\star)^2}\frac{\gamma}{N}\\&- \frac{f'''(x^\star)\sigma^2}{12f''(x^\star)} \frac{(H-1)(5H-1)}{H}\gamma^2 +O_H\!\left(\frac{\gamma^2}{N}+\gamma^3\right). \end{aligned} $$ Hence client averaging suppresses the leading $O(\gamma/N)$ bias but does not remove the client-independent $O(\gamma^2)$ component when its coefficient is nonzero. The mechanism is indirect: although the direct control contribution cancels pathwise in the linear global average, the controls still alter within-round local trajectories and their second moments. Fresh gradient noise and persistent control fluctuations therefore generate local second-moment corrections that nonquadratic curvature converts into stationary mean bias. The coefficient vanishes for quadratic objectives. Numerical experiments are consistent with the predicted coefficient, its persistence as client count increases, and the stated joint remainder. The result is restricted to the one-dimensional homogeneous fixed-$H$ setting.
