[Paper Review] REMIND Your Neural Network to Prevent Catastrophic Forgetting

Tyler L. Hayes, Kushal Kafle+3 more|arXiv (Cornell University)|Oct 6, 2019

Multimodal Machine Learning ApplicationsComputer Science87 references22 citations

TL;DR

REMIND is a brain-inspired online learning framework that prevents catastrophic forgetting in neural networks by replaying compressed, quantized feature representations instead of raw pixels. Using Product Quantization for efficient memory indexing, REMIND achieves state-of-the-art performance on ImageNet in streaming learning, outperforming existing methods by 21.9% while training in under 12 hours—demonstrating robustness and generalization to tasks like streaming Visual Question Answering.

ABSTRACT

People learn throughout life. However, incrementally updating conventional neural networks leads to catastrophic forgetting. A common remedy is replay, which is inspired by how the brain consolidates memory. Replay involves fine-tuning a network on a mixture of new and old instances. While there is neuroscientific evidence that the brain replays compressed memories, existing methods for convolutional networks replay raw images. Here, we propose REMIND, a brain-inspired approach that enables efficient replay with compressed representations. REMIND is trained in an online manner, meaning it learns one example at a time, which is closer to how humans learn. Under the same constraints, REMIND outperforms other methods for incremental class learning on the ImageNet ILSVRC-2012 dataset. We probe REMIND's robustness to data ordering schemes known to induce catastrophic forgetting. We demonstrate REMIND's generality by pioneering online learning for Visual Question Answering (VQA).

Motivation & Objective

• To address catastrophic forgetting in continual online learning, where neural networks overwrite prior knowledge when trained on non-iid data streams.
• To develop a biologically plausible learning method inspired by hippocampal indexing and memory replay in the brain.
• To enable efficient, memory-constrained online learning using compressed representations rather than raw image replay.
• To extend the approach to new tasks such as streaming Visual Question Answering (VQA), previously unexplored in online learning.
• To establish new baselines and evaluation protocols for streaming VQA and demonstrate strong performance on CLEVR and TDIUC datasets.

Proposed method

• REMIND uses Product Quantization (PQ) to compress and store hidden feature maps (e.g., from ResNet-18) for later replay, enabling efficient memory indexing.
• The model performs online learning by updating the network one example at a time, simulating human-like streaming learning.
• During training, REMIND fine-tunes the network on a mixture of current data and randomly sampled compressed representations from past experiences.
• The replay buffer is updated via random replacement, maintaining a fixed-size memory of compressed features without significant computational overhead.
• For VQA, REMIND adapts by replaying compressed visual-linguistic features, preserving multimodal representations across streaming inputs.
• The method avoids reliance on task labels at test time, making it robust to real-world data ordering that induces forgetting.

Experimental results

Research questions

• RQ1 Can replay of compressed, high-level feature representations mitigate catastrophic forgetting more effectively than raw pixel replay in online learning?
• RQ2 How does REMIND perform compared to batch and streaming baselines on ImageNet under strict memory and compute constraints?
• RQ3 Can REMIND generalize to multimodal tasks like streaming Visual Question Answering (VQA) without architectural modification?
• RQ4 Does REMIND maintain robustness when data is ordered in ways that induce catastrophic forgetting, such as class-incremental or instance-incremental schemes?
• RQ5 Can REMIND achieve performance close to offline full-batch training while training in under 12 hours, unlike batch methods requiring 65+ hours?

Key findings

• REMIND achieved 99.5% top-5 accuracy on ImageNet in the streaming setting, outperforming the best streaming baseline by 21.9% and being only 1.9% below the best batch model.
• On the CORe50 dataset, REMIND outperformed regularization-based methods like EWC, MAS, and A-GEM, even when task labels were withheld, demonstrating robustness to forgetting.
• REMIND reduced training time to under 12 hours on ImageNet, compared to 65 hours for BiC, a state-of-the-art batch method.
• In streaming VQA, REMIND achieved strong performance on CLEVR and TDIUC, establishing new baselines and demonstrating generalization beyond image classification.
• REMIND’s performance remained stable across different data ordering schemes, including class-ordered and instance-ordered streams, indicating resilience to forgetting-inducing sequences.
• The use of compressed representations (via PQ) enabled REMIND to store significantly more experiences in the same memory budget than methods relying on raw pixel replay.