[Paper Review] Continual Learning Through Synaptic Intelligence

Friedemann Zenke, Ben Poole+1 more|arXiv (Cornell University)|Mar 13, 2017

Domain Adaptation and Few-Shot Learning21 references68 citations

TL;DR

The paper introduces intelligent synapses that accumulate per-synapse importance during training and apply a online quadratic consolidation penalty to protect important weights, enabling continual learning with reduced forgetting and maintained efficiency.

ABSTRACT

While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.

Motivation & Objective

• Motivate and address catastrophic forgetting in neural networks when learning from non-stationary data streams.
• Propose a synapse-level mechanism that online estimates each parameter's importance to past tasks.
• Develop a local, online consolidation penalty that protects important synapses while allowing unimportant ones to adapt to new tasks.
• Show that online synaptic intelligence stabilizes learning across multiple tasks with competitive performance and scalability.

Proposed method

• Define a per-synapse importance measure ωμk as the running sum of gradient × parameter update for each synapse during task μ.
• Construct a quadratic surrogate loss ˜Lμ that preserves the minimum and the trajectory of past tasks by penalizing changes to important synapses with Ωμk = sumν<μωνk(∆νk)2 + ξ.
• Blend the current task loss Lμ with the consolidation term using a strength parameter c in ˜Lμ = Lμ + c Σk Ωμk(˜θk − θk)^2.
• Update ωk online during training and accumulate Ωμk at the end of each task, resetting ωk afterward to enable continual learning across tasks.
• Draw connections to Hessian-based measures, showing that in certain cases the consolidated importances align with Hessian structure, especially in low-rank or diagonal scenarios.

Experimental results

Research questions

• RQ1 How can synaptic-level dynamics be used to prevent catastrophic forgetting in continual learning?
• RQ2 Can online, per-synapse importance estimation combined with a local consolidation penalty maintain performance on old tasks while learning new ones?
• RQ3 How does the proposed Synaptic Intelligence framework compare to existing methods like Elastic Weight Consolidation (EWC) in terms of effectiveness and online computation?
• RQ4 What theoretical relationships exist between the online importance measure and the curvature (Hessian) of the loss landscape?
• RQ5 Is the approach scalable to larger datasets and more complex architectures beyond simple MNIST splits?

Key findings

• Consolidation with per-synapse importance dramatically reduces forgetting on split and permuted MNIST while preserving learning efficiency.
• On split MNIST, networks with consolidation maintain high average accuracy across tasks versus substantial forgetting without consolidation.
• On permuted MNIST, consolidation achieves performance close to joint training and compares favorably with EWC.
• In CIFAR-10/100 splits, consolidation yields better generalization and protects older memories across multiple tasks compared to no consolidation.
• Correlation analyses show consolidation reduces cross-task coupling of important weights, especially in deeper layers, mitigating interference.
• The online, per-synapse approach provides a scalable alternative to offline Fisher-based measures and aligns with low-rank Hessian intuition for memory consolidation.