[Paper Review] Bridging the Gap: Enhancing the Utility of Synthetic Data via Post-Processing Techniques

A. Lampis, Eugenio Lomurno+1 more|arXiv (Cornell University)|May 17, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces Gap Filler (GaFi), a post-processing pipeline that enhances synthetic data utility for image classification by applying Dynamic Sample Filtering, Dynamic Dataset Recycle, and the Expansion Trick to Generative Adversarial Networks (GANs). GaFi reduces the accuracy gap between synthetic and real data to just 2.03%, 1.78%, and 3.99% on Fashion-MNIST, CIFAR-10, and CIFAR-100, respectively, setting a new state of the art in Classification Accuracy Score (CAS).

ABSTRACT

Acquiring and annotating suitable datasets for training deep learning models is challenging. This often results in tedious and time-consuming efforts that can hinder research progress. However, generative models have emerged as a promising solution for generating synthetic datasets that can replace or augment real-world data. Despite this, the effectiveness of synthetic data is limited by their inability to fully capture the complexity and diversity of real-world data. To address this issue, we explore the use of Generative Adversarial Networks to generate synthetic datasets for training classifiers that are subsequently evaluated on real-world images. To improve the quality and diversity of the synthetic dataset, we propose three novel post-processing techniques: Dynamic Sample Filtering, Dynamic Dataset Recycle, and Expansion Trick. In addition, we introduce a pipeline called Gap Filler (GaFi), which applies these techniques in an optimal and coordinated manner to maximise classification accuracy on real-world data. Our experiments show that GaFi effectively reduces the gap with real-accuracy scores to an error of 2.03%, 1.78%, and 3.99% on the Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets, respectively. These results represent a new state of the art in Classification Accuracy Score and highlight the effectiveness of post-processing techniques in improving the quality of synthetic datasets.

Motivation & Objective

• Address the persistent performance gap between classifiers trained on synthetic data and those trained on real data, despite high perceptual quality of generated samples.
• Overcome limitations in synthetic data utility caused by distributional discrepancies and lack of diversity, even when using state-of-the-art GANs.
• Develop a model-agnostic post-processing pipeline that improves synthetic data quality without modifying the generative model’s architecture or training process.
• Maximize Classification Accuracy Score (CAS) on real-world test sets by optimizing synthetic data through coordinated post-processing techniques.
• Demonstrate that post-processing can close the utility gap to near-real performance, enabling synthetic data to serve as a viable alternative to real data in downstream tasks.

Proposed method

• Propose Dynamic Sample Filtering: use a pre-trained real-data classifier to filter out low-confidence synthetic samples, improving label consistency and data quality.
• Introduce Dynamic Dataset Recycle: iteratively retrain the generator on filtered synthetic data and re-evaluate using the real classifier, enabling progressive refinement of the synthetic dataset.
• Develop the Expansion Trick: augment synthetic data by generating additional samples from latent space interpolation, increasing diversity and coverage of the data distribution.
• Design the GaFi pipeline as a coordinated workflow integrating the three techniques, with adaptive hyperparameters and recycling intervals to maximize CAS.
• Apply the pipeline to BigGAN-Deep-generated data across Fashion-MNIST, CIFAR-10, and CIFAR-100, using the same evaluation protocol as prior work for fair comparison.
• Optimize pipeline hyperparameters via ablation studies, including filtering thresholds and recycling frequency, to achieve optimal CAS performance.

Figure 1: Overview of the Gap Filler (GaFi) pipeline.

Experimental results

Research questions

• RQ1 To what extent can post-processing techniques reduce the classification accuracy gap between synthetic and real data when using GAN-generated images?
• RQ2 How do Dynamic Sample Filtering, Dynamic Dataset Recycle, and the Expansion Trick individually and collectively improve the utility of synthetic datasets for downstream classification tasks?
• RQ3 Can a model-agnostic post-processing pipeline achieve state-of-the-art CAS performance without modifying the generative model’s architecture or training process?
• RQ4 What is the optimal configuration of post-processing techniques in terms of filtering thresholds, recycling frequency, and data augmentation to maximize CAS?
• RQ5 Does the GaFi pipeline enable synthetic data to approach or even surpass real-data performance in classification tasks?

Key findings

• The GaFi pipeline reduces the accuracy gap between synthetic and real data to 2.03% on Fashion-MNIST, 1.78% on CIFAR-10, and 3.99% on CIFAR-100, representing a new state of the art in Classification Accuracy Score (CAS).
• The Expansion Trick significantly improves CAS by increasing data diversity, with the most effective results observed when combined with Dynamic Sample Filtering.
• GaFi achieves higher CAS than the previous state of the art even when using only one generator, outperforming the prior method that required six generators.
• The pipeline improves CAS by 5.33% on Fashion-MNIST, 6.09% on CIFAR-10, and 14.21% on CIFAR-100 compared to the synthetic baseline (BigGAN-Deep without post-processing).
• The final CAS results of 94.03% on Fashion-MNIST, 93.20% on CIFAR-10, and 71.95% on CIFAR-100 are within 2% of real-data accuracy, demonstrating near-parity with real-data training.
• The combination of Dynamic Sample Filtering and Dynamic Dataset Recycle leads to a substantial shift in CAS performance, with reduced standard deviation across runs, indicating improved stability and generalization.

Figure 2: The results of the CAS metric obtained using the Expansion Trick technique. The plots compare unfiltered and filtered datasets (filtering threshold: 0.0).