[Paper Review] MatchingGAN: Matching-based Few-shot Image Generation

Yan Hong, Li Niu+2 more|arXiv (Cornell University)|Mar 7, 2020

Generative Adversarial Networks and Image Synthesis31 references4 citations

TL;DR

MatchingGAN proposes a few-shot image generation framework that combines adversarial learning with a matching mechanism to generate diverse, realistic images from only a few reference images per category. By learning interpolation coefficients via feature matching between random vectors and conditional images, it enables high-quality, diverse generation without fine-tuning on unseen categories.

ABSTRACT

To generate new images for a given category, most deep generative models require abundant training images from this category, which are often too expensive to acquire. To achieve the goal of generation based on only a few images, we propose matching-based Generative Adversarial Network (GAN) for few-shot generation, which includes a matching generator and a matching discriminator. Matching generator can match random vectors with a few conditional images from the same category and generate new images for this category based on the fused features. The matching discriminator extends conventional GAN discriminator by matching the feature of generated image with the fused feature of conditional images. Extensive experiments on three datasets demonstrate the effectiveness of our proposed method.

Motivation & Objective

• Address the challenge of generating realistic images for unseen categories when only a few training images are available.
• Overcome limitations of existing few-shot image generation methods, such as low image quality, lack of diversity, or dependency on fine-tuning.
• Leverage matching-based feature alignment to learn meaningful interpolation coefficients between reference images and random noise.
• Design a novel GAN architecture that integrates matching procedures into both generator and discriminator for improved feature consistency and realism.
• Enable zero-shot generalization to unseen categories without retraining or fine-tuning on test data.

Proposed method

• Propose a matching generator that projects random vectors and conditional images into a shared feature space, then computes similarity scores as interpolation coefficients to fuse features.
• Use a matching discriminator that not only distinguishes real from fake images but also matches the discriminative features of generated images with the fused features of conditional images.
• Train the generator and discriminator end-to-end using adversarial loss and feature matching loss, with hyperparameters λr and λm controlling the trade-off between reconstruction and realism.
• Employ shared encoder parameters (Eϕ and Eψ) for efficiency and stability, with skip connections between encoder and decoder to preserve multi-level features.
• Utilize a matching procedure to learn interpolation weights rather than relying on random sampling, improving feature fusion quality.
• Apply the trained model at inference to generate diverse images for unseen categories by feeding new random vectors into the matching generator.

Experimental results

Research questions

• RQ1 Can a matching-based mechanism improve the quality and diversity of few-shot image generation compared to random interpolation?
• RQ2 Does integrating feature matching into both generator and discriminator lead to better alignment between generated images and conditional reference images?
• RQ3 Can the proposed MatchingGAN generalize to unseen categories without fine-tuning, leveraging only a few reference images?
• RQ4 How do design choices such as number of skip connections and encoder sharing affect generation performance and diversity?
• RQ5 What is the optimal trade-off between adversarial loss and feature matching loss in few-shot image generation?

Key findings

• MatchingGAN achieves a test accuracy of 40.95% on the VGGFace dataset with K=3, outperforming the random coefficient baseline (38.12%) and demonstrating the benefit of learned interpolation.
• Setting the feature matching loss weight λm=0 degrades performance significantly, with FID increasing from 108.56 to 111.40 and IS dropping from 8.32 to 7.56, proving the importance of feature matching.
• Using three skip connections improves IS (9.14) and reduces FID (106.12), but harms low-data classification accuracy (34.12%), indicating a trade-off between realism and diversity.
• The model with shared encoders (Eϕ and Eψ) achieves 40.95% accuracy, while separate encoders yield only 40.98% with slightly better FID (107.98), showing minimal benefit from increased parameters.
• The ablation study confirms that λr=0.1 provides the best balance between reconstruction and adversarial loss, with λr=1.0 and λr=10.0 yielding worse results in all metrics.
• Qualitative results on Omniglot, EMNIST, and VGGFace show that generated images are diverse, realistic, and semantically consistent with the reference images, even for unseen categories.