[Paper Review] DeltaGAN: Towards Diverse Few-shot Image Generation with Sample-Specific Delta

Yan Hong, Li Niu+2 more|arXiv (Cornell University)|Jul 21, 2022

Generative Adversarial Networks and Image Synthesis79 references4 citations

TL;DR

DeltaGAN proposes a novel few-shot image generation framework that enhances diversity by learning sample-specific deltas—transformation patterns between support images—using a reconstruction subnetwork and a generation subnetwork. By introducing an adversarial delta matching loss, the method generates diverse, high-quality images from few-shot inputs, outperforming prior approaches on six benchmark datasets.

ABSTRACT

Learning to generate new images for a novel category based on only a few images, named as few-shot image generation, has attracted increasing research interest. Several state-of-the-art works have yielded impressive results, but the diversity is still limited. In this work, we propose a novel Delta Generative Adversarial Network (DeltaGAN), which consists of a reconstruction subnetwork and a generation subnetwork. The reconstruction subnetwork captures intra-category transformation, i.e., delta, between same-category pairs. The generation subnetwork generates sample-specific delta for an input image, which is combined with this input image to generate a new image within the same category. Besides, an adversarial delta matching loss is designed to link the above two subnetworks together. Extensive experiments on six benchmark datasets demonstrate the effectiveness of our proposed method. Our code is available at https://github.com/bcmi/DeltaGAN-Few-Shot-Image-Generation.

Motivation & Objective

• To address the limited diversity in existing few-shot image generation methods despite strong reconstruction performance.
• To model intra-category variations by learning transformation deltas between support images of the same category.
• To generate diverse, realistic images by applying sample-specific deltas to input support images.
• To unify reconstruction and generation subnetworks through an adversarial delta matching loss for improved feature alignment.

Proposed method

• The method employs a reconstruction subnetwork that learns intra-category deltas between pairs of support images from the same class.
• A generation subnetwork learns to produce sample-specific deltas for a given input image, which are then applied to generate new images.
• An adversarial delta matching loss is introduced to align the generated deltas with the real deltas from the reconstruction subnetwork.
• The framework combines the input image with the generated delta to synthesize new, diverse images within the same category.
• The model is trained end-to-end on few-shot image generation benchmarks using both reconstruction and adversarial objectives.
• The architecture is evaluated on six standard few-shot image generation datasets to validate performance and diversity.

Experimental results

Research questions

• RQ1 Can learning sample-specific deltas improve diversity in few-shot image generation without sacrificing reconstruction quality?
• RQ2 How effective is adversarial delta matching in aligning generated deltas with real intra-category transformations?
• RQ3 To what extent does DeltaGAN outperform existing state-of-the-art methods in generating diverse, realistic images from few examples?
• RQ4 Can the reconstruction and generation subnetworks be effectively synchronized through an adversarial loss on delta representations?

Key findings

• DeltaGAN achieves state-of-the-art performance on six benchmark datasets in few-shot image generation, demonstrating superior diversity and image quality.
• The adversarial delta matching loss significantly improves the alignment between generated and real deltas, enhancing generation fidelity.
• The method generates more diverse samples compared to prior approaches, as measured by Inception Score and FID metrics.
• The reconstruction subnetwork effectively captures intra-category variations, enabling accurate delta learning across support image pairs.
• The generation subnetwork produces realistic, category-consistent images by applying learned, sample-specific deltas to input images.
• Ablation studies confirm that both the delta matching loss and the dual subnetwork design are essential for performance gains.