[Paper Review] Continual Learning with Self-Organizing Maps

Pouya Bashivan, Martin Schrimpf+4 more|arXiv (Cornell University)|Apr 19, 2019

Domain Adaptation and Few-Shot Learning10 references4 citations

TL;DR

This paper proposes SOMLP, a memoryless continual learning method that uses Self-Organizing Maps (SOMs) to dynamically route inputs to task-specific network regions without requiring explicit task boundaries or memory buffers. By clustering inputs into task contexts unsupervised and masking network activations accordingly, SOMLP achieves performance on par with state-of-the-art methods like GEM and EWC on MNIST permutations and rotations, especially excelling in low-memory regimes.

ABSTRACT

Despite remarkable successes achieved by modern neural networks in a wide range of applications, these networks perform best in domain-specific stationary environments where they are trained only once on large-scale controlled data repositories. When exposed to non-stationary learning environments, current neural networks tend to forget what they had previously learned, a phenomena known as catastrophic forgetting. Most previous approaches to this problem rely on memory replay buffers which store samples from previously learned tasks, and use them to regularize the learning on new ones. This approach suffers from the important disadvantage of not scaling well to real-life problems in which the memory requirements become enormous. We propose a memoryless method that combines standard supervised neural networks with self-organizing maps to solve the continual learning problem. The role of the self-organizing map is to adaptively cluster the inputs into appropriate task contexts - without explicit labels - and allocate network resources accordingly. Thus, it selectively routes the inputs in accord with previous experience, ensuring that past learning is maintained and does not interfere with current learning. Out method is intuitive, memoryless, and performs on par with current state-of-the-art approaches on standard benchmarks.

Motivation & Objective

• Address catastrophic forgetting in continual learning without relying on memory buffers or explicit task boundaries.
• Develop a scalable, memory-efficient alternative to existing experience replay and regularization-based continual learning methods.
• Enable neural networks to adaptively allocate resources to new tasks while preserving knowledge from previous tasks.
• Leverage unsupervised representation learning via SOMs to infer task structure from input data distributions.
• Demonstrate that self-organizing maps can serve as an effective, label-free mechanism for dynamic resource allocation in continual learning.

Proposed method

• Integrate a Self-Organizing Map (SOM) layer in parallel with a fully connected feedforward network to learn input distribution structure without supervision.
• Train the SOM using competitive learning: for each input batch, identify the Best Matching Unit (BMU) and update it and its topological neighbors using a decaying learning rate α and neighborhood radius σ.
• Compute a task-specific output mask Γ using L2 distance between input and SOM weights, scaled by a hyperparameter ε, to gate the fully connected layer’s activations.
• Use the mask Γ to selectively activate only the network nodes relevant to the current input’s task context, preventing interference with previously learned tasks.
• Pretrain SOM weights on unlabeled data from all tasks (SOMLP_m) or on the full MNIST training set (SOMLP_s) to improve initialization and generalization.
• Decay learning rate α and neighborhood radius σ over time using an exponential decay rate τ to stabilize training and avoid overfitting.

Experimental results

Research questions

• RQ1 Can a self-organizing map be used to implicitly infer task boundaries in continual learning without explicit task labels?
• RQ2 Does a memoryless approach using unsupervised clustering via SOMs outperform memory-based methods like EWC and GEM in low-memory settings?
• RQ3 How well can SOMLP maintain performance across sequential tasks when no experience replay or Fisher information is used?
• RQ4 To what extent do SOMLP’s learned masks reflect the underlying similarity structure between tasks (e.g., rotations vs. permutations)?
• RQ5 Can unsupervised feature learning via SOMs enable effective dynamic resource allocation in neural networks during continual learning?

Key findings

• SOMLP_m (pretrained on 10% of all tasks) achieved 85.4% ± 1.8 accuracy on MNIST-permutations and 79.1% ± 0.8 on MNIST-rotations, outperforming EWC and matching GEM in some settings.
• In low-memory regimes (e.g., 8 memory slots), SOMLP_m surpassed all baseline methods, including EWC and GEM, demonstrating superior scalability.
• SOMLP_s (pretrained on full MNIST) performed worse than SOMLP_m but still outperformed EWC on MNIST-rotations (59.1% vs. 49.8%) and matched EWC on MNIST-permutations.
• GEM’s performance degraded significantly with reduced memory size, while EWC’s performance was inconsistent across datasets, suggesting sensitivity to memory buffer size.
• The learned SOM masks in MNIST-rotations shared nodes between similar tasks (e.g., small rotation differences), while using separate nodes for dissimilar tasks, indicating effective task clustering.
• Mean performance over sequential tasks remained stable for GEM and SOMLP_m, while EWC and SOMLP_s showed performance degradation, indicating forgetting under low-capacity or unstable conditions.