Search papers, labs, and topics across Lattice.
This paper introduces spectral gradient orthogonalization as a post-processing technique for differentially private training, addressing the issue of isotropic Gaussian noise corrupting gradient directions. By leveraging the low-rank structure of gradients in vision models, the method recovers directional signal without incurring additional privacy costs, leading to significant improvements in model accuracy. Empirical results show that this approach yields up to a 20.9% accuracy increase on WRN-28-10 and a 14.9% increase on ResNet-18, particularly in high-SNR regimes associated with larger batch sizes.
Spectral gradient orthogonalization can boost model accuracy by over 20% in differentially private training, but only under the right conditions.
Differentially private training adds isotropic Gaussian noise to clipped gradients, corrupting every singular direction equally. In vision models, where spatial correlation concentrates gradient energy into a low-rank subspace, most of this noise falls in directions that carry little signal. Spectral gradient orthogonalization via polar decomposition is introduced as a post-processing step that recovers directional signal from the noisy gradient's low-rank structure at zero additional privacy cost. A phase transition governs the utility of this approach: orthogonalization improves accuracy only when the per-direction spectral signal-to-noise ratio (SNR) suffices for singular vector recovery; in low-SNR regimes, the directional bias of the gradient is replaced by a nearly random orthogonal update, and the transformation is harmful. The recovery threshold is determined by the spectral gap of the gradient and is surpassed at large batch sizes. Empirically, the benefit scales with model capacity: spectral orthogonalization achieves a +20.9% improvement over DP-SGD on WRN-28-10 (B = 4096) and +14.9% on ResNet-18, while reducing inter-run variance by a factor of two to three. In the fine-tuning regime, spectral orthogonalization matches the stability of DP-Adam while maintaining a first-order memory footprint. Combining spectral with temporal denoising yields 50.3% on CIFAR-10 (epsilon = 4), the highest accuracy in any tested configuration. These gains are specific to moderate-to-high-SNR regimes such as large-batch training of higher-capacity models. Small-batch or low-SNR settings are better served by DP-SGD or temporal denoising.
