Skip to main content
QUICK REVIEW

[Paper Review] Effective Data Augmentation With Diffusion Models

Brandon Trabucco, Kyle G. Doherty|arXiv (Cornell University)|Feb 7, 2023
Domain Adaptation and Few-Shot LearningComputer Science81 citations
TL;DR

The paper introduces DA-Fusion, a diffusion-model–based data augmentation method that edits real images semantically to create diverse, task-relevant synthetic data for few-shot classification, while addressing leakage from pretrained models.

ABSTRACT

Data augmentation is one of the most prevalent tools in deep learning, underpinning many recent advances, including those from classification, generative models, and representation learning. The standard approach to data augmentation combines simple transformations like rotations and flips to generate new images from existing ones. However, these new images lack diversity along key semantic axes present in the data. Current augmentations cannot alter the high-level semantic attributes, such as animal species present in a scene, to enhance the diversity of data. We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained text-to-image diffusion models. Our method edits images to change their semantics using an off-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples. We evaluate our approach on few-shot image classification tasks, and on a real-world weed recognition task, and observe an improvement in accuracy in tested domains.

Motivation & Objective

  • Motivation: standard augmentations lack semantic diversity and fail to alter high-level attributes.
  • Goal: develop a flexible, off-the-shelf diffusion-based augmentation that applies to any image and improves few-shot classification.
  • Aim: balance real and synthetic data and generalize to unseen concepts with minimal domain-specific tuning.

Proposed method

  • Use text-to-image diffusion models to edit images semantically via new embeddings inserted into the model's text encoder.
  • Adapt diffusion models to unseen concepts by learning new embeddings with Textual Inversion for a handful of labeled examples.
  • Splice real images into the diffusion process (SDEdit) to generate synthetic images guided by learned embeddings.
  • Balance real and synthetic data in training batches with a probability parameter to mix real and augmented images.
  • Introduce randomized augmentation intensity by varying the insertion timestep t0 during image splicing to increase diversity.
  • Implement leakage-prevention strategies: model-centric (erase class concepts from model weights) and data-centric (omit class names from prompts).

Experimental results

Research questions

  • RQ1Can diffusion-based, semantically editing augmentations improve few-shot classification on datasets where concepts are outside the diffusion model vocabulary?
  • RQ2How do leakage-prevention strategies affect the effectiveness of diffusion-based data augmentation?
  • RQ3Does randomness in augmentation intensity contribute to performance gains, and how robust is the method to real/synthetic data balance?
  • RQ4 Is the approach effective across multiple domains (Pascal VOC, COCO, leafy spurge weed dataset)?

Key findings

  • DA-Fusion improves few-shot classification accuracy across three datasets, with gains up to about 10 percentage points over standard augmentation baselines.
  • Model-centric leakage prevention still benefits, showing up to +5 percentage points in Pascal and COCO domains.
  • Data-centric leakage prevention yields larger gains, up to about +10 percentage points, indicating reliance on or interaction with model priors.
  • Randomizing augmentation intensity (varying t0) consistently improves performance over fixed intensity.
  • DA-Fusion is robust to the balance between real and synthetic data (alpha and M) with modest sensitivity.
  • A contributed weed dataset (leafy spurge) demonstrates DA-Fusion’s ability to handle unseen concepts outside the diffusion model vocabulary.

Read this paper

Open the full text in the Nubint reader and ask about any passage as you read.

Read this paper on Nubint

This review was generated by AI from the original paper. Please check the paper before citing it.