Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for differentially private (DP) learning that circumvents the challenges of iterative optimization in parameter space by utilizing a hypernetwork. Instead of relying on privatized gradients, the method maps a private dataset to target model parameters through a single injection of noise into a low-dimensional representation of the dataset. The results demonstrate that this approach not only maintains higher utility under a fixed privacy budget compared to DP-SGD but also achieves lower FID scores in practical applications like LoRA fine-tuning of diffusion models.
Injecting noise just once into a low-dimensional representation can significantly enhance model utility in differentially private learning.
Differentially private (DP) training of neural networks is often hindered by the large amount of noise required by gradient-based methods such as DP-SGD, which repeatedly inject high-dimensional noise in parameter space throughout training. In this paper, we propose a new framework for DP learning that avoids iterative optimization in parameter space. Instead of updating the target model using privatized gradients, we employ a hypernetwork trained on public datasets to map a private dataset to the parameters of the target model. Specifically, each example is embedded into a low-dimensional representation, the embeddings are aggregated and perturbed to obtain a DP dataset embedding, and the hypernetwork generates the target model parameters from this noisy embedding. Because privacy noise is injected only once into a low-dimensional dataset representation, our approach can significantly reduce the adverse effect of noise. We theoretically show in a synthetic setting that, under a fixed privacy budget, models produced by our approach achieve higher utility than those trained with DP-SGD. Moreover, we apply our approach to LoRA fine-tuning of diffusion models and show that it achieves lower FID than LoRA models trained with DP-SGD and other public-data-guided methods.
