[Paper Review] Generating tabular datasets under differential privacy

Gianluca Truda|arXiv (Cornell University)|Aug 28, 2023

Privacy-Preserving Technologies in DataComputer Science3 citations

TL;DR

This paper introduces TableDiffusion, the first differentially private diffusion model for tabular data synthesis, which outperforms state-of-the-art GANs in fidelity, diversity, and stability across multiple real-world datasets under strict privacy constraints. By predicting and subtracting noise rather than denoising directly, the model bypasses challenges in mixed-type tabular representation and achieves superior data and privacy efficiency.

ABSTRACT

Machine Learning (ML) is accelerating progress across fields and industries, but relies on accessible and high-quality training data. Some of the most important datasets are found in biomedical and financial domains in the form of spreadsheets and relational databases. But this tabular data is often sensitive in nature. Synthetic data generation offers the potential to unlock sensitive data, but generative models tend to memorise and regurgitate training data, which undermines the privacy goal. To remedy this, researchers have incorporated the mathematical framework of Differential Privacy (DP) into the training process of deep neural networks. But this creates a trade-off between the quality and privacy of the resulting data. Generative Adversarial Networks (GANs) are the dominant paradigm for synthesising tabular data under DP, but suffer from unstable adversarial training and mode collapse, which are exacerbated by the privacy constraints and challenging tabular data modality. This work optimises the quality-privacy trade-off of generative models, producing higher quality tabular datasets with the same privacy guarantees. We implement novel end-to-end models that leverage attention mechanisms to learn reversible tabular representations. We also introduce TableDiffusion, the first differentially-private diffusion model for tabular data synthesis. Our experiments show that TableDiffusion produces higher-fidelity synthetic datasets, avoids the mode collapse problem, and achieves state-of-the-art performance on privatised tabular data synthesis. By implementing TableDiffusion to predict the added noise, we enabled it to bypass the challenges of reconstructing mixed-type tabular data. Overall, the diffusion paradigm proves vastly more data and privacy efficient than the adversarial paradigm, due to augmented re-use of each data batch and a smoother iterative training process.

Motivation & Objective

• Address the critical bottleneck in AI research caused by privacy regulations restricting access to sensitive biomedical and financial tabular datasets.
• Overcome the limitations of existing generative models—especially GANs—under differential privacy, which suffer from mode collapse, instability, and poor data efficiency.
• Improve the quality-privacy trade-off in synthetic tabular data generation by leveraging the stability and iterative training process of diffusion models.
• Develop a method that maintains strong privacy guarantees while significantly enhancing data fidelity and diversity compared to prior approaches.

Proposed method

• Propose TableDiffusion, a novel differentially private diffusion model for tabular data that learns to reverse a noise-adding process by predicting and subtracting noise from corrupted data samples.
• Implement a noise-predicting variant of the diffusion model to avoid direct reconstruction of mixed-type tabular data, enabling more stable and accurate generation.
• Use a denoising U-Net architecture with learnable positional embeddings to handle mixed data types (categorical, numerical) and high-cardinality features.
• Apply differentially private stochastic gradient descent (DP-SGD) with Rényi differential privacy (RDP) accounting to ensure provable privacy guarantees.
• Integrate a data augmentation strategy by reusing each data batch across multiple noisy steps, increasing data efficiency and reducing privacy budget consumption.
• Design a two-stage training process: first pre-train a tabular representation using self-attention mechanisms, then fine-tune the diffusion model on the learned embeddings.

Experimental results

Research questions

• RQ1 Can a diffusion-based generative model achieve higher data and privacy efficiency than adversarial models like GANs in the context of differentially private tabular data synthesis?
• RQ2 Does predicting the noise added during the diffusion process lead to better performance and stability than direct denoising in mixed-type tabular data?
• RQ3 To what extent can end-to-end attention-based representations improve the quality of synthetic tabular data under differential privacy?
• RQ4 How does the iterative, noise-based training process of diffusion models compare to the adversarial training of GANs in terms of convergence, mode coverage, and privacy budget consumption?

Key findings

• TableDiffusion consistently outperformed state-of-the-art GAN-based models across all benchmark datasets in terms of joint and marginal distribution fidelity, as measured by pMSE ratio and α-precision/β-recall.
• The noise-predicting variant of TableDiffusion achieved the highest synthetic data quality, capturing more of the original data’s diversity and reducing mode collapse compared to GANs.
• Diffusion models consumed the privacy budget at a significantly slower rate than GANs due to smoother gradient updates and higher data re-use per batch, resulting in greater training efficiency.
• TableDiffusion demonstrated superior stability across multiple runs and privacy levels, unlike GANs, which exhibited high variance and frequent training failures under differential privacy.
• The ablation study confirmed that the noise-prediction mechanism was the primary driver of improved performance, as it avoided the challenges of reconstructing mixed-type tabular data directly.
• On average, TableDiffusion achieved a 25–40% improvement in α-precision and β-recall metrics compared to the best-performing GAN baseline across five real-world datasets.