[Paper Review] P3GM: Private High-Dimensional Data Release via Privacy Preserving Phased Generative Model

Shun Takagi, Tsubasa Takahashi+2 more|arXiv (Cornell University)|Jun 22, 2020

Privacy-Preserving Technologies in Data36 references4 citations

TL;DR

P3GM proposes a two-phase differentially private generative model for high-dimensional data release, using privacy-preserving PCA and EM in the encoding phase followed by DP-SGD in the decoding phase. It achieves superior sample quality and utility in downstream tasks compared to state-of-the-art methods, especially in high-dimensional settings with minimal noise degradation.

ABSTRACT

How can we release a massive volume of sensitive data while mitigating privacy risks? Privacy-preserving data synthesis enables the data holder to outsource analytical tasks to an untrusted third party. The state-of-the-art approach for this problem is to build a generative model under differential privacy, which offers a rigorous privacy guarantee. However, the existing method cannot adequately handle high dimensional data. In particular, when the input dataset contains a large number of features, the existing techniques require injecting a prohibitive amount of noise to satisfy differential privacy, which results in the outsourced data analysis meaningless. To address the above issue, this paper proposes privacy-preserving phased generative model (P3GM), which is a differentially private generative model for releasing such sensitive data. P3GM employs the two-phase learning process to make it robust against the noise, and to increase learning efficiency (e.g., easy to converge). We give theoretical analyses about the learning complexity and privacy loss in P3GM. We further experimentally evaluate our proposed method and demonstrate that P3GM significantly outperforms existing solutions. Compared with the state-of-the-art methods, our generated samples look fewer noises and closer to the original data in terms of data diversity. Besides, in several data mining tasks with synthesized data, our model outperforms the competitors in terms of accuracy.

Motivation & Objective

• Address the challenge of releasing high-dimensional sensitive data while preserving privacy and utility.
• Overcome the limitations of existing differentially private generative models that inject excessive noise in high-dimensional settings.
• Develop a privacy-preserving generative model that maintains data diversity and fidelity under strict differential privacy constraints.
• Enable practical deployment of private data synthesis for real-world high-dimensional datasets such as images and sensor traces.
• Balance privacy budget allocation between encoding and decoding phases to optimize model performance and robustness.

Proposed method

• Employ a two-phase training process: first, a privacy-preserving encoding phase using differentially private PCA (DP-PCA) and differentially private EM (DP-EM) to learn a low-dimensional latent representation.
• Use the Gaussian mechanism instead of the Wishart mechanism for DP-PCA to ensure rigorous differential privacy guarantees.
• Apply differentially private stochastic gradient descent (DP-SGD) in the decoding phase to train a variational autoencoder (VAE) on the latent space.
• Leverage RDP-based privacy composition to more accurately account for privacy loss across multiple mechanisms, reducing overall epsilon budget.
• Integrate a privacy budget allocation strategy that dynamically balances noise injection between encoding and decoding components.
• Use total variation distance and AUROC scores to evaluate the fidelity of 2-way marginal distributions and classification utility of synthetic data.

Experimental results

Research questions

• RQ1 Can a two-phase differentially private generative model effectively handle high-dimensional data while minimizing noise injection?
• RQ2 How does P3GM compare to existing DP-VAE and DP-GM methods in terms of sample quality and downstream task accuracy?
• RQ3 What is the optimal privacy budget allocation between the encoding and decoding phases for maximizing model utility?
• RQ4 Does the use of RDP-based privacy composition lead to tighter privacy accounting than standard composition theorems?
• RQ5 To what extent does P3GM preserve complex data dependencies and marginal distributions in high-dimensional datasets?

Key findings

• P3GM generates synthetic samples that are visually closer to real data and contain less noise than DP-VAE, DP-GM, and PrivBayes, especially on MNIST.
• In classification tasks, P3GM outperforms all competitors, achieving an average AUROC of 0.3867 on the Adult dataset and 0.3029 on ISOLET, significantly outperforming Ryan’s algorithm in utility despite Ryan’s superior 2-way marginal accuracy.
• P3GM maintains strong performance across high-dimensional datasets, with AUROC scores stable from 5 to 100 dimensions, while other methods degrade rapidly.
• The model achieves a total variational distance of 0.2411 on the Kaggle Credit dataset, outperforming DP-GM (0.1345) and PrivBayes (0.0082) in utility for high-dimensional data.
• RDP-based privacy composition results in a smaller epsilon value than baseline zCDP and MA methods, enabling tighter privacy accounting.
• Optimal privacy budget allocation for P3GM lies between 0.1 and 0.3 for the encoding phase, with performance peaking in this range, suggesting a non-uniform allocation improves utility.