[Paper Review] On the Usefulness of Synthetic Tabular Data Generation

Dionysis Manousakas, Sergül Aydöre|arXiv (Cornell University)|Jun 27, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper evaluates the utility of synthetic tabular data for machine learning training across four use cases: data sharing, summarization, augmentation, and class balancing. Despite advances in generative modeling, the authors find no consistent improvement in downstream ML performance when using synthetic data, concluding that current methods do not reliably boost model efficacy beyond training on real data alone.

ABSTRACT

Despite recent advances in synthetic data generation, the scientific community still lacks a unified consensus on its usefulness. It is commonly believed that synthetic data can be used for both data exchange and boosting machine learning (ML) training. Privacy-preserving synthetic data generation can accelerate data exchange for downstream tasks, but there is not enough evidence to show how or why synthetic data can boost ML training. In this study, we benchmarked ML performance using synthetic tabular data for four use cases: data sharing, data augmentation, class balancing, and data summarization. We observed marginal improvements for the balancing use case on some datasets. However, we conclude that there is not enough evidence to claim that synthetic tabular data is useful for ML training.

Motivation & Objective

• To assess whether synthetic tabular data can improve downstream machine learning performance across practical use cases.
• To evaluate the effectiveness of generative models in data augmentation, class balancing, data summarization, and data sharing scenarios.
• To determine if synthetic data can match or exceed the performance of real data in training models for tabular benchmarks.
• To identify gaps in current synthetic data evaluation by simulating real-world deployment scenarios often missing in existing literature.
• To provide empirical evidence on the limitations of state-of-the-art tabular generative models in enhancing ML training efficiency and accuracy.

Proposed method

• Benchmarked four use cases: data sharing (TS-TR), summarization (synthetic data replacing real data), data augmentation (TA-TR), and class balancing (conditional generation for minority classes).
• Used two generative modeling approaches: deep generative models (DGMs) trained from scratch and fine-tuned pre-trained large language models (LLMs).
• Trained generative models on 10%, 50%, and 100% of real data subsets to assess scalability and data efficiency.
• Evaluated downstream performance using AutoGluon classifiers on real test sets after training on synthetic or augmented data.
• Compared results against real data baselines and SMOTE for class balancing, using standard predictive metrics (e.g., AUC, accuracy).
• Focused on model-agnostic, generic tabular learning tasks to assess general applicability across datasets.

Figure 1: ( Synthetic data quality experiment ) Predictive performance of a downstream AutoGluon classifier, which uses synthetic training data sampled from generative models that have been trained on increasing subsets of the real data. The number of synthetic training datapoints increases along th

Experimental results

Research questions

• RQ1 Can synthetic tabular data achieve comparable or better downstream ML performance than real data in data sharing scenarios?
• RQ2 To what extent can synthetic data compress statistical information from real data for efficient summarization?
• RQ3 Does data augmentation with synthetic samples improve predictive accuracy in data-scarce or imbalanced tabular learning tasks?
• RQ4 Can conditional generation of synthetic data effectively balance class distributions and improve model fairness or performance?
• RQ5 Are current state-of-the-art generative models for tabular data consistently effective across diverse real-world use cases?

Key findings

• In data sharing (TS-TR), synthetic data performance was comparable to real data, but no consistent improvement was observed across datasets.
• For data summarization, synthetic data achieved lower downstream performance than equivalent real data volumes, with only marginal gains over random subsets.
• In data augmentation, synthetic data did not consistently improve model performance; results were often worse or on par with real data alone.
• For class balancing, synthetic data only marginally outperformed real data and SMOTE on two out of four datasets, with no significant gains in AUC or accuracy.
• Overall, no consistent evidence was found that synthetic tabular data boosts ML training efficacy across the evaluated use cases.
• The study highlights a critical gap in the literature: real-world deployment simulations are often missing, and current methods fail to reliably enhance model performance beyond real data.

Figure 2: ( Data summarization experiment ) Predictive performance of a downstream AutoGluon classifier which uses synthetic training data sampled from a generative model that has been trained on the full real dataset. The number of synthetic training data increases along the $x$ -axis.