Search papers, labs, and topics across Lattice.
This paper introduces SAGE, a noise-aware shrinkage method for differentially private zeroth-order optimization (DP-ZO) that adaptively adjusts the weight of privatized model updates based on their estimated signal quality. By stabilizing the estimate of signal energy and applying a bounded shrinkage factor, SAGE effectively mitigates the negative impact of noise-dominated updates, leading to improved model utility. Experimental results show that SAGE consistently outperforms existing aggregation-based DP-ZO methods across various large language models while maintaining memory efficiency.
Noise-aware shrinkage can enhance the utility of differentially private fine-tuning, outperforming traditional methods without compromising privacy or efficiency.
Differentially private zeroth-order optimization (DP-ZO) enables memory-efficient private fine-tuning of large language models using only forward evaluations. Existing aggregation-based DP-ZO methods reconstruct model updates at a fixed scale, ignoring that the strength of useful signals varies throughout training. Consequently, noise-dominated updates may receive excessive weight and degrade model utility. To address this issue, we propose SAGE, a noise-aware shrinkage method that adaptively attenuates privatized estimates according to their estimated signal quality. SAGE subtracts the known Gaussian noise variance from the observed second moment to estimate the underlying signal energy, stabilizes this estimate through temporal tracking, and compares its current signal-to-noise level with a warm-up reference to derive a bounded shrinkage factor. As pure post-processing, SAGE requires neither additional privacy budget nor model queries and introduces only constant additional state. Our theoretical analysis shows that shrinkage reduces the quadratic update-risk term faster than the linear descent term, preserving useful descent while limiting the influence of noise-dominated updates. Experiments on RoBERTa-large, OPT-1.3B, and OPT-6.7B demonstrate that SAGE outperforms existing baselines in most settings under the same privacy budgets while preserving the forward-only memory efficiency of DP-ZO.
