[Paper Review] Utility Theory of Synthetic Data Generation

Shirong Xu, Will Wei Sun+1 more|arXiv (Cornell University)|May 17, 2023

Auction Theory and ApplicationsDecision Sciences3 citations

TL;DR

This paper establishes a statistical utility theory for synthetic data generation by defining a general utility metric as the absolute difference in generalization error between models trained on synthetic and original data. It derives analytical bounds showing that model specification correctness enables utility convergence even without perfect feature fidelity, offering theoretical guidance for designing effective synthetic data algorithms in regression and classification tasks.

ABSTRACT

Synthetic data algorithms are widely employed in industries to generate artificial data for downstream learning tasks. While existing research primarily focuses on empirically evaluating utility of synthetic data, its theoretical understanding is largely lacking. This paper bridges the practice-theory gap by establishing relevant utility theory in a statistical learning framework. It considers two utility metrics: generalization and ranking of models trained on synthetic data. The former is defined as the generalization difference between models trained on synthetic and on real data. By deriving analytical bounds for this utility metric, we demonstrate that the synthetic feature distribution does not need to be similar as that of real data for ensuring comparable generalization of synthetic models, provided proper model specifications in downstream learning tasks. The latter utility metric studies the relative performance of models trained on synthetic data. In particular, we discover that the distribution of synthetic data is not necessarily similar as the real one to ensure consistent model comparison. Interestingly, consistent model comparison is still achievable even when synthetic responses are not well generated, as long as downstream models are separable by a generalization gap. Finally, extensive experiments on non-parametric models and deep neural networks have been conducted to validate these theoretical findings.

Motivation & Objective

• To address the lack of theoretical understanding in how synthetic data algorithms affect downstream model generalization performance.
• To develop a general, quantifiable utility metric based on the difference in generalization error between original and synthetic data.
• To identify critical conditions under which the utility metric converges to zero, ensuring synthetic data preserves model performance.
• To establish sufficient conditions for synthetic data algorithms to maintain consistent model ranking by generalization performance.
• To validate theoretical findings empirically using non-parametric models and deep neural networks.

Proposed method

• Defining a utility metric as the absolute difference in generalization error between models trained on original and synthetic datasets.
• Deriving analytical bounds for the utility metric in regression and classification, decomposing it into four components: estimation error, synthetic feature quality, regression function estimation, and model specification.
• Using chi-square divergence and fidelity level concepts to characterize the similarity between original and synthetic feature distributions.
• Establishing convergence conditions for the utility metric under various assumptions, including model specification correctness and feature fidelity.
• Applying the bounds to three examples: Gaussian mixtures, bounded density ratios, and (V,d)-fidelity level, demonstrating robustness across distributions.
• Validating theoretical results through extensive experiments on non-parametric and deep learning models.

Figure 1 : The architecture for generating and evaluating synthetic data in supervised learning.

Experimental results

Research questions

• RQ1 Under what conditions does the generalization error difference between models trained on synthetic and original data converge to zero?
• RQ2 How does model specification correctness affect the necessity of perfect feature fidelity for utility convergence?
• RQ3 What properties must a synthetic data algorithm possess to preserve the ranking of model generalization performance?
• RQ4 How do the components of the utility metric—estimation error, feature quality, function estimation, and model specification—influence overall utility?
• RQ5 Can theoretical bounds on utility be validated empirically across diverse model architectures and data types?

Key findings

• The utility metric converges to zero if the synthetic features have perfect fidelity and the relationship between features and responses is well-approximated by the synthetic data algorithm.
• Surprisingly, perfect feature fidelity is not required for utility convergence if the model specification in the downstream task is correct, highlighting the importance of model structure over distributional fidelity.
• The analytical bounds decompose utility into four key components, providing a roadmap for improving synthetic data algorithms by targeting specific error sources.
• For distributions with bounded density ratios, the chi-square divergence between original and synthetic distributions can be bounded using the (V,d)-fidelity level, ensuring utility convergence under mild conditions.
• Empirical validation confirms that theoretical bounds hold across non-parametric models and deep neural networks, supporting the robustness of the proposed framework.
• The framework establishes sufficient conditions under which the ranking of model generalization performance based on synthetic data matches that from original data, enabling reliable model comparison.

Figure 2 : An illustrative example for consistent model comparison when the synthetic and original distributions are not the same.