[Paper Review] Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control

Inyoung Paik, Sangjun Oh+2 more|arXiv (Cornell University)|Jul 31, 2019

Domain Adaptation and Few-Shot Learning27 references4 citations

TL;DR

This paper proposes Neuron-level Plasticity Control (NPC), a novel continual learning method that mitigates catastrophic forgetting by dynamically adjusting learning rates at the neuron level based on each neuron's importance, rather than constraining individual connection weights. NPC significantly outperforms connection-level consolidation methods like EWC, MAS, and SI on iMNIST and iCIFAR100, achieving average accuracies of 99.63% and 71.87% respectively, while eliminating the need to store task-specific parameter values, thus reducing memory overhead.

ABSTRACT

To address the issue of catastrophic forgetting in neural networks, we propose a novel, simple, and effective solution called neuron-level plasticity control (NPC). While learning a new task, the proposed method preserves the knowledge for the previous tasks by controlling the plasticity of the network at the neuron level. NPC estimates the importance value of each neuron and consolidates important extit{neurons} by applying lower learning rates, rather than restricting individual connection weights to stay close to certain values. The experimental results on the incremental MNIST (iMNIST) and incremental CIFAR100 (iCIFAR100) datasets show that neuron-level consolidation is substantially more effective compared to the connection-level consolidation approaches.

Motivation & Objective

• To address catastrophic forgetting in continual learning, where sequential training overwrites previously learned knowledge in deep neural networks.
• To overcome the limitations of connection-level consolidation methods that restrict individual weights and require storing task-specific parameter values.
• To improve knowledge retention by consolidating important neurons through dynamic learning rate adjustment based on neuron-level importance.
• To reduce memory and computational overhead by storing only a single importance value per neuron, independent of the number of tasks.
• To explore neuron-level plasticity as a more effective and efficient alternative to weight-based consolidation in deep learning.

Proposed method

• NPC estimates the importance of each neuron using the magnitude of its gradient during task learning, treating neurons as the unit of consolidation rather than individual weights.
• It controls plasticity by applying lower learning rates to neurons deemed more important for previous tasks, thereby preserving their learned representations.
• The method uses a single importance value per neuron, avoiding the need to store multiple sets of optimized weights across tasks.
• Neuron importance is computed as the average absolute gradient magnitude over the training process, reflecting the neuron's contribution to previous task performance.
• The loss function incorporates a regularization term that penalizes changes in high-importance neurons, effectively consolidating them.
• The approach is compatible with standard backpropagation and can be applied to feedforward and convolutional networks without architectural changes.

Experimental results

Research questions

• RQ1 Can neuron-level plasticity control outperform traditional connection-level consolidation methods in mitigating catastrophic forgetting?
• RQ2 Does controlling plasticity at the neuron level lead to better retention of knowledge from previous tasks compared to weight-level regularization?
• RQ3 Can NPC maintain high performance across multiple tasks without storing task-specific parameter values, thus reducing memory overhead?
• RQ4 How does the distribution of activation changes across neurons correlate with their importance under NPC?
• RQ5 Is NPC effective on diverse datasets such as iMNIST and iCIFAR100, especially when task complexity increases?

Key findings

• On the iMNIST dataset, NPC achieved an average validation accuracy of 99.63% (±0.02), significantly outperforming EWC (95.42%), MAS (98.76%), SI (97.24%), and SSL (96.87%).
• On the iCIFAR100 dataset, NPC achieved an average accuracy of 71.87% (±0.65), surpassing EWC (64.57%), MAS (65.54%), SI (57.91%), and SSL (65.16%), with only a slight deficit on Task 5.
• The average activation change of the most important 10% of neurons under NPC was only 0.094, compared to 0.383 for all neurons, indicating strong preservation of critical representations.
• The average activation change of the least important 10% of neurons was 0.667, significantly higher than the overall average, showing that unimportant neurons are more adaptable to new tasks.
• Without consolidation, no meaningful correlation existed between neuron importance and activation change, confirming that NPC effectively targets the right neurons for preservation.
• NPC requires no task-specific parameter storage, maintaining constant memory usage regardless of the number of tasks, unlike connection-level methods that scale linearly with task count.