[Paper Review] DreamDA: Generative Data Augmentation with Diffusion Models

Yunxiang Fu, Chaoqi Chen+2 more|arXiv (Cornell University)|Mar 19, 2024

Big Data Technologies and ApplicationsDecision Sciences3 citations

TL;DR

DreamDA proposes a novel generative data augmentation framework that leverages pre-trained diffusion models to generate diverse, high-fidelity images by perturbing the reverse diffusion process of real training images, while introducing a self-training paradigm (AMST) to generate reliable pseudo-labels. It achieves consistent accuracy gains across four tasks and five datasets, outperforming strong baselines by over 7.4% when training from scratch on natural image datasets.

ABSTRACT

The acquisition of large-scale, high-quality data is a resource-intensive and time-consuming endeavor. Compared to conventional Data Augmentation (DA) techniques (e.g. cropping and rotation), exploiting prevailing diffusion models for data generation has received scant attention in classification tasks. Existing generative DA methods either inadequately bridge the domain gap between real-world and synthesized images, or inherently suffer from a lack of diversity. To solve these issues, this paper proposes a new classification-oriented framework DreamDA, which enables data synthesis and label generation by way of diffusion models. DreamDA generates diverse samples that adhere to the original data distribution by considering training images in the original data as seeds and perturbing their reverse diffusion process. In addition, since the labels of the generated data may not align with the labels of their corresponding seed images, we introduce a self-training paradigm for generating pseudo labels and training classifiers using the synthesized data. Extensive experiments across four tasks and five datasets demonstrate consistent improvements over strong baselines, revealing the efficacy of DreamDA in synthesizing high-quality and diverse images with accurate labels. Our code will be available at https://github.com/yunxiangfu2001/DreamDA.

Motivation & Objective

• To address the limitations of conventional data augmentation and existing generative DA methods, which often fail to bridge domain gaps or lack diversity.
• To enable high-fidelity, diverse image synthesis that adheres to the original data distribution using pre-trained diffusion models.
• To solve the label inconsistency problem in generated data by introducing a robust self-training framework with consistency and confidence constraints.
• To demonstrate consistent performance gains across diverse vision tasks and datasets using both training-from-scratch and fine-tuning settings.

Proposed method

• Perturbs the reverse diffusion process by injecting Gaussian noise into the U-Net bottleneck layer at each denoising step, enabling diverse yet realistic image generation.
• Uses real training images as seeds to guide the diffusion process, ensuring generated samples remain within the original data distribution.
• Introduces Asymmetric Multi-Head Self-Training (AMST), which enforces consistency and confidence constraints across multiple classifier heads to generate reliable pseudo-labels.
• Applies a consistency regularization loss to encourage the model to produce consistent predictions for similar data points, even when labels are uncertain.
• Employs a multi-head architecture with shared features and separate heads to improve robustness and generalization of pseudo-label predictions.
• Combines real and synthetic data during training, enabling end-to-end learning with minimal computational overhead.

Experimental results

Research questions

• RQ1 Can perturbing the reverse diffusion process of real images using noise injection in the U-Net bottleneck generate diverse, high-fidelity images that conform to the original data distribution?
• RQ2 How can label inconsistency in generated images—caused by semantic shifts during diffusion—be effectively mitigated during training?
• RQ3 Does the proposed Asymmetric Multi-Head Self-Training (AMST) framework outperform standard pseudo-labeling methods in terms of accuracy and robustness?
• RQ4 What is the optimal location and noise scale for perturbing the diffusion process to balance image fidelity and diversity?
• RQ5 Can DreamDA achieve consistent improvements across diverse vision tasks and datasets, including both training-from-scratch and fine-tuning scenarios?

Key findings

• DreamDA improves model accuracy by over 41% on natural image datasets when training from scratch, compared to using only real data.
• On pre-trained models, DreamDA boosts accuracy by 4% over the original real dataset, demonstrating strong transferability.
• DreamDA outperforms the strongest diffusion-based DA baseline by 7.4% when training from scratch on natural image datasets.
• The method achieves state-of-the-art FID and MMD scores, indicating superior distributional similarity and diversity of generated images.
• Ablation studies confirm that both latent perturbation and AMST are essential, with performance dropping by 5.6% and 2.4% respectively when removed.
• Optimal noise scale for perturbation is 3, and injecting noise at the U-Net bottleneck layer yields the best performance, outperforming perturbations at other locations.