Search papers, labs, and topics across Lattice.
This paper introduces DP-SAPF, a saliency-aware parameter selection method for differentially private image synthesis, which selectively applies LoRA to the most salient parameters based on noisy gradient magnitudes computed on sensitive data. By focusing DP-SGD training on these salient parameters, DP-SAPF mitigates noise accumulation and parameter collapse, leading to improved utility and fidelity of synthetic images. Experiments across four sensitive image datasets demonstrate that DP-SAPF achieves better performance with fewer computational resources compared to exhaustive LoRA fine-tuning.
Stop wasting compute on irrelevant parameters: selectively fine-tuning only the *salient* parameters of public models dramatically improves differentially private image synthesis.
Differentially private (DP) image synthesis generates images that preserve the statistical characteristics of a sensitive dataset, enabling sensitive data analysis and usage while providing rigorous guarantees of privacy leakage. Existing methods fine-tune public models using DP Stochastic Gradient Descent (DP-SGD) on sensitive images to generate synthetic images. But full fine-tuning public models on sensitive images is computationally expensive, because current public models typically contain a large number of parameters. Recent work proposes heuristically using Low-Rank Adaptation (LoRA) on all attention-layer parameters of public models to reduce the number of trainable parameters. However, we argue that exhaustive LoRA coverage across all attention-layer parameters is suboptimal in a DP setting, as it leads to noise accumulation and collapse during private training. To address this issue, we propose DP-SAPF, which uses a saliency-aware strategy to identify specific target parameters for LoRA training under DP. DP-SAPF is inspired by the fact that larger gradients signify higher saliency, indicating that these parameters are most critical for the DP learning. Specifically, we feed the sensitive images into public models, compute gradients, and add noise to the gradients to satisfy DP. Then, DP-SAPF identifies the most salient parameters, those exhibiting high gradient magnitudes on sensitive images, for DP fine-tuning. Experiments on four sensitive image datasets show that DP-SAPF improves the utility and fidelity of synthetic images while requiring fewer computational resources than fine-tuning methods without parameter selection.
