[Paper Review] GAN Memory with No Forgetting

Yulai Cong, Miaoyun Zhao+3 more|arXiv (Cornell University)|Jun 13, 2020

Domain Adaptation and Few-Shot Learning98 references4 citations

TL;DR

This paper proposes GAN Memory, a lifelong learning framework that uses style modulation of a pre-trained GAN to generate realistic samples from new, perceptually distant data distributions without forgetting previous tasks. By applying modified FiLM and AdaIN layers to adapt the generator and discriminator, the method achieves high-fidelity generative replay with minimal forgetting, outperforming baselines on CIFAR-10 and ImageNet benchmarks while enabling parameter compression and knowledge sharing across tasks.

ABSTRACT

As a fundamental issue in lifelong learning, catastrophic forgetting is directly caused by inaccessible historical data; accordingly, if the data (information) were memorized perfectly, no forgetting should be expected. Motivated by that, we propose a GAN memory for lifelong learning, which is capable of remembering a stream of datasets via generative processes, with \emph{no} forgetting. Our GAN memory is based on recognizing that one can modulate the "style" of a GAN model to form perceptually-distant targeted generation. Accordingly, we propose to do sequential style modulations atop a well-behaved base GAN model, to form sequential targeted generative models, while simultaneously benefiting from the transferred base knowledge. The GAN memory -- that is motivated by lifelong learning -- is therefore itself manifested by a form of lifelong learning, via forward transfer and modulation of information from prior tasks. Experiments demonstrate the superiority of our method over existing approaches and its effectiveness in alleviating catastrophic forgetting for lifelong classification problems. Code is available at https://github.com/MiaoyunZhao/GANmemory_LifelongLearning.

Motivation & Objective

• To address catastrophic forgetting in lifelong learning by memorizing data distributions via generative modeling instead of storing raw data.
• To develop a scalable, privacy-preserving generative replay mechanism that avoids the limitations of blurry or low-quality samples in existing methods.
• To enable continual learning with high-dimensional data (e.g., images) over long sequences of tasks without performance degradation.
• To achieve parameter efficiency and knowledge reuse across tasks through compressible, task-specific style parameters.

Proposed method

• Leverages a well-behaved pre-trained GAN as a base model, with its generator and discriminator adapted via modified style modulation techniques.
• Introduces mFiLM and mAdaFM to modulate fully connected and convolutional layers respectively, enabling style transfer to new, perceptually distant domains.
• Applies sequential style modulation to generate task-specific GANs while preserving the base model’s knowledge, forming a lifelong generative memory.
• Employs low-rank approximation and parameter sharing to compress style parameters, reducing model size without performance loss.
• Uses the GAN memory as a realistic generative replay mechanism to alleviate forgetting in downstream classification tasks.
• Generalizes the framework to a conditional variant to support class-conditional generation and improved downstream performance.

Experimental results

Research questions

• RQ1 Can a single pre-trained GAN be effectively adapted to generate realistic samples from multiple, perceptually distinct data distributions without forgetting previous tasks?
• RQ2 How can style modulation techniques be extended to both generator and discriminator components of a GAN for effective lifelong learning?
• RQ3 To what extent can the number of task-specific style parameters be reduced through compression while maintaining generation quality and no forgetting?
• RQ4 Does the proposed GAN memory framework outperform existing generative replay and coreset-based methods in alleviating catastrophic forgetting on complex datasets like CIFAR-10 and ImageNet?
• RQ5 Can the learned style parameters be shared or reused across tasks, enabling efficient lifelong learning with growing generative capacity?

Key findings

• The GAN memory achieves test accuracy within 1.5% of the joint-training upper bound on CIFAR-10 with 6 tasks, demonstrating strong performance in continual learning.
• On ImageNet-based butterfly classification tasks, the compressed GAN memory reduces style parameter count by up to 97.3% (from 10.6M to 0.3M) while maintaining competitive FID scores.
• The FID score of the compressed GAN memory (27.67 on D1) is comparable to the naive version (28.89), confirming effective compression without performance degradation.
• Parameter sharing across tasks reduces the number of new parameters needed for later tasks by up to 97%, showing strong knowledge reusability.
• The method achieves state-of-the-art performance on lifelong learning benchmarks, outperforming existing generative replay and coreset-based approaches.
• The framework enables realistic, high-fidelity sample generation across diverse domains, with no catastrophic forgetting observed even after 6 sequential tasks.