[Paper Review] Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks

Youngjin Kim, Minjung Kim+1 more|arXiv (Cornell University)|Feb 15, 2018

Generative Adversarial Networks and Image Synthesis20 citations

TL;DR

The paper proposes memoryGAN, an unsupervised GAN framework that integrates a learnable memory network to alleviate structural discontinuity in latent space and discriminator forgetting during adversarial training. By enabling both generators and discriminators to access shared memory, the model improves representation learning and training stability, achieving state-of-the-art inception scores on CIFAR10 without optimization tricks.

ABSTRACT

We propose an approach to address two issues that commonly occur during training of unsupervised GANs. First, since GANs use only a continuous latent distribution to embed multiple classes or clusters of data, they often do not correctly handle the structural discontinuity between disparate classes in a latent space. Second, discriminators of GANs easily forget about past generated samples by generators, incurring instability during adversarial training. We argue that these two infamous problems of unsupervised GAN training can be largely alleviated by a learnable memory network to which both generators and discriminators can access. Generators can effectively learn representation of training samples to understand underlying cluster distributions of data, which ease the structure discontinuity problem. At the same time, discriminators can better memorize clusters of previously generated samples, which mitigate the forgetting problem. We propose a novel end-to-end GAN model named memoryGAN, which involves a memory network that is unsupervisedly trainable and integrable to many existing GAN models. With evaluations on multiple datasets such as Fashion-MNIST, CelebA, CIFAR10, and Chairs, we show that our model is probabilistically interpretable, and generates realistic image samples of high visual fidelity. The memoryGAN also achieves the state-of-the-art inception scores over unsupervised GAN models on the CIFAR10 dataset, without any optimization tricks and weaker divergences.

Motivation & Objective

• To address structural discontinuity in the latent space of unsupervised GANs caused by a single continuous latent distribution for multiple data clusters.
• To mitigate discriminator forgetting during adversarial training, where discriminators fail to retain knowledge of previously generated samples.
• To design a learnable, end-to-end trainable memory network that enhances both generator and discriminator performance in unsupervised GANs.
• To improve the probabilistic interpretability and visual fidelity of generated samples across diverse datasets.
• To achieve state-of-the-art inception scores on CIFAR10 without relying on optimization tricks or weaker divergence losses.

Proposed method

• Introduces a memory network that stores and retrieves representations of real and generated data samples, accessible to both generator and discriminator.
• Uses a differentiable memory read/write mechanism to allow the generator to attend to stored data representations for better cluster-aware generation.
• Employs a memory-augmented discriminator that maintains a memory of past generated samples to reduce forgetting and stabilize training.
• Integrates the memory network into existing GAN architectures as a plug-in module, enabling compatibility with various GAN variants.
• Trains the entire model end-to-end using standard GAN objectives, with memory updates performed via backpropagation.
• Leverages memory-based attention mechanisms to improve the generator's understanding of underlying data cluster distributions.

Experimental results

Research questions

• RQ1 Can a shared memory network reduce structural discontinuity in the latent space of unsupervised GANs?
• RQ2 To what extent does memory-augmented training reduce discriminator forgetting during adversarial training?
• RQ3 How does memoryGAN improve the probabilistic interpretability and visual quality of generated samples?
• RQ4 Can memoryGAN achieve state-of-the-art performance on standard benchmarks like CIFAR10 without architectural or optimization tricks?
• RQ5 How does the memory network enhance the stability and convergence of GAN training across diverse datasets?

Key findings

• memoryGAN achieves state-of-the-art inception score on CIFAR10 among unsupervised GAN models, without using optimization tricks or weaker divergence losses.
• The model generates high-fidelity images with improved visual quality on Fashion-MNIST, CelebA, and Chairs datasets.
• The memory network enables better representation learning, reducing latent space discontinuity between distinct data clusters.
• Discriminators in memoryGAN show reduced forgetting, as evidenced by more stable training dynamics and consistent performance over time.
• The model is probabilistically interpretable, with memory access patterns reflecting meaningful data cluster structures.
• The memory network is successfully integrated into various GAN architectures, demonstrating broad compatibility and effectiveness.