[Paper Review] Effective Data Augmentation with Multi-Domain Learning GANs

Shin’ya Yamaguchi, Sekitoshi Kanai+1 more|arXiv (Cornell University)|Dec 25, 2019

Image Processing Techniques and Applications33 references4 citations

TL;DR

This paper proposes Domain Fusion (DF), a multi-domain learning GAN-based data augmentation method that improves model generalization on low-resource datasets by jointly training on a target domain and an external outer domain. By leveraging shared GAN parameters, DF generates high-fidelity, diverse samples that outperform conventional data augmentation and fine-tuned GANs, achieving state-of-the-art accuracy on CIFAR-100, FGVC-Aircraft, and Indoor Scene Recognition with only 5,000 target samples.

ABSTRACT

For deep learning applications, the massive data development (e.g., collecting, labeling), which is an essential process in building practical applications, still incurs seriously high costs. In this work, we propose an effective data augmentation method based on generative adversarial networks (GANs), called Domain Fusion. Our key idea is to import the knowledge contained in an outer dataset to a target model by using a multi-domain learning GAN. The multi-domain learning GAN simultaneously learns the outer and target dataset and generates new samples for the target tasks. The simultaneous learning process makes GANs generate the target samples with high fidelity and variety. As a result, we can obtain accurate models for the target tasks by using these generated samples even if we only have an extremely low volume target dataset. We experimentally evaluate the advantages of Domain Fusion in image classification tasks on 3 target datasets: CIFAR-100, FGVC-Aircraft, and Indoor Scene Recognition. When trained on each target dataset reduced the samples to 5,000 images, Domain Fusion achieves better classification accuracy than the data augmentation using fine-tuned GANs. Furthermore, we show that Domain Fusion improves the quality of generated samples, and the improvements can contribute to higher accuracy.

Motivation & Objective

• To address the high cost and limited scalability of data collection and annotation in deep learning.
• To overcome the limitations of traditional data augmentation, which cannot generate truly unseen or diverse samples from low-resource datasets.
• To improve model generalization on small target datasets by leveraging knowledge from an external outer domain through GAN-based generation.
• To develop a method that generates high-fidelity, diverse samples even with minimal target data, avoiding performance degradation from poor-quality GAN outputs.
• To systematically evaluate the impact of outer domain selection and sample quality on downstream classification performance.

Proposed method

• Proposes Domain Fusion (DF), a conditional GAN framework that jointly trains on both a target dataset and an external outer dataset to generate realistic, class-conditional samples.
• Employs multi-domain learning GANs with shared parameters to transfer knowledge from the outer domain to the target domain during training.
• Introduces a novel metric combining Fréchet Inception Distance (FID) and multi-scale structural similarity (MS-SSIM) to select the most relevant outer dataset based on domain relevance and sample diversity.
• Applies discriminator rejection sampling (DRS) extended for conditional GANs to filter out low-quality, unrealistic generated samples.
• Combines DF-generated samples with conventional data augmentation to further improve model accuracy.
• Uses conditional GANs to generate class-specific samples, ensuring that generated data aligns with target task semantics.

Experimental results

Research questions

• RQ1 Can multi-domain learning GANs generate higher-quality and more diverse samples than fine-tuned GANs when training data is extremely limited?
• RQ2 How does the choice of outer dataset affect the performance of data augmentation in low-data regimes?
• RQ3 To what extent does the quality of generated samples correlate with downstream classification accuracy?
• RQ4 Can combining DF with conventional data augmentation yield better performance than conventional augmentation alone?
• RQ5 Does multi-domain learning preserve knowledge from the outer domain better than fine-tuning-based approaches?

Key findings

• On CIFAR-100 with only 5,000 training images, Domain Fusion achieved higher classification accuracy than both conventional data augmentation and fine-tuned GANs (TGANs).
• DF improved top-1 accuracy on Food Containers and Household Furnitures superclasses by 1.25 and 1.21, respectively, outperforming TGANs and CGANs.
• For the People superclass, DF with Pascal-VOC as outer dataset achieved a 1.20 improvement rate, significantly higher than TGAN’s 1.07, indicating better knowledge transfer.
• DF generated samples with lower FID (125.6) and higher Inception Score (3.85) for the People class compared to TGAN (FID: 135.5, IS: 3.29), confirming higher sample quality.
• The combination of DF and conventional data augmentation outperformed conventional augmentation alone, demonstrating complementary benefits.
• The study confirmed a strong correlation between the quality of generated samples (measured by FID and IS) and the resulting classification accuracy.