[Paper Review] GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators

Dingfan Chen, Tribhuvanesh Orekondy+1 more|arXiv (Cornell University)|Jun 15, 2020

Privacy-Preserving Technologies in Data46 references30 citations

TL;DR

GS-WGAN trains a DP GAN by selectively sanitizing generator gradients under a Wasserstein objective, improving high-dimensional data generation in centralized and federated settings.

ABSTRACT

The wide-spread availability of rich data has fueled the growth of machine learning applications in numerous domains. However, growth in domains with highly-sensitive data (e.g., medical) is largely hindered as the private nature of data prohibits it from being shared. To this end, we propose Gradient-sanitized Wasserstein Generative Adversarial Networks (GS-WGAN), which allows releasing a sanitized form of the sensitive data with rigorous privacy guarantees. In contrast to prior work, our approach is able to distort gradient information more precisely, and thereby enabling training deeper models which generate more informative samples. Moreover, our formulation naturally allows for training GANs in both centralized and federated (i.e., decentralized) data scenarios. Through extensive experiments, we find our approach consistently outperforms state-of-the-art approaches across multiple metrics (e.g., sample quality) and datasets.

Motivation & Objective

• Motivate privacy-preserving data publishing for high-dimensional data.
• Propose a gradient-sanitized DP-GAN (GS-WGAN) that selectively sanitizes generator gradients.
• Leverage Wasserstein loss with gradient penalty to bound gradient sensitivity and improve utility.
• Extend GS-WGAN to federated learning with provable DP guarantees.

Proposed method

• Use DP-SGD-like gradient sanitization applied selectively to the generator parameters.
• Adopt a Wasserstein GAN objective with a gradient penalty to keep discriminator gradients Lipschitz.
• Clip and add noise to the generator upstream gradient while leveraging the chain rule to minimize information loss.
• Exploit post-processing and subsampling to amplify privacy, enabling user-level DP in federated settings.
• Train multiple discriminators on disjoint subsets in a federated-like setup to reduce privacy costs.

Experimental results

Research questions

• RQ1 Can selective gradient sanitization of only the generator yield better utility under DP than full DP-SGD in GANs?
• RQ2 Does a Wasserstein objective with gradient penalty stabilize gradient norms to ease privacy budgeting?
• RQ3 Can GS-WGAN be effectively applied in federated, decentralized data settings with provable DP guarantees?
• RQ4 How does subsampling affect privacy-utility trade-offs in DP GAN training?

Key findings

• GS-WGAN achieves higher sample quality than state-of-the-art DP GAN baselines on MNIST and Fashion-MNIST (IS improvements and lower FID).
• The Wasserstein objective with gradient penalty yields bounded gradient norms, reducing clipping-induced utility loss.
• Selective sanitization to generator parameters enables DP training with competitive or superior downstream task performance.
• Federated GS-WGAN provides DP guarantees with lower privacy cost and reduced communication compared to Fed-Avg GAN.
• Subsampling and warm-start strategies further improve privacy-utility trade-offs and robustness to noise.