Search papers, labs, and topics across Lattice.
This paper introduces a parameter clipping strategy for Nonparametric Variational Differential Privacy (NVDP) to improve privacy guarantees and utility in language models. The clipping method is derived from minimizing the Rényi Divergence (RD) upper bound, providing constraints on posterior mean, variance, and mixture weight parameters. Empirical results on an NVIB-based model demonstrate that the clipped model achieves tighter RD bounds, indicating stronger privacy, and simultaneously improves performance on downstream tasks compared to an unconstrained baseline.
Tighter privacy guarantees and higher utility in language models are simultaneously achievable via a principled parameter clipping strategy for Nonparametric Variational Differential Privacy.
The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the Rényi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.
