[Paper Review] Selfless Sequential Learning

Rahaf Aljundi, Marcus Rohrbach+1 more|arXiv (Cornell University)|Jun 14, 2018

Domain Adaptation and Few-Shot LearningComputer Science40 references53 citations

TL;DR

The paper introduces SLNID, a representation-based regularizer that promotes sparse, locally inhibited, and task-importance-aware neuron activations to improve lifelong learning with fixed model capacity.

ABSTRACT

Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them. To achieve Selfless Sequential Learning we study different regularization strategies and activation functions. We find that imposing sparsity at the level of the representation (i.e.~neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity. In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition. It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks. As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations, our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain. We combine our novel regularizer, with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network. We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement %over alternative regularizers we studied on diverse datasets.

Motivation & Objective

• Motivate lifelong learning under fixed model capacity where future tasks should be accommodated.
• Investigate representation sparsity vs. parameter sparsity for reducing interference across tasks.
• Propose a novel regularizer (SLNID) that implements local neural inhibition with neuron importance discounting.
• Demonstrate that representation-based sparsity plus SLNID improves performance across diverse datasets and baselines.

Proposed method

• Propose a regularization framework combining MAS/EWC style importance preservation with a representation-based sparsity objective on activations.
• Introduce Sparse coding through Local Neural Inhibition and Discounting (SLNID), which penalizes simultaneous activations of nearby neurons (local inhibition).
• Extend SLNID with neuronal importance discounting to protect previously important neurons from inhibition when they are relevant to the current task.
• Formulate SLNID as a local, Gaussian-weighted inhibition term over hidden activations, modulated by previous task neuron importance (alpha_i).
• Demonstrate compatibility by integrating SLNID with MAS (and also with EWC) and evaluating on permuted MNIST, CIFAR-100, and Tiny ImageNet.

Experimental results

Research questions

• RQ1 Does imposing sparsity in neuron activations (representation sparsity) yield better lifelong learning performance than parameter sparsity in a fixed-capacity model?
• RQ2 Can local (not global) neural inhibition combined with neuron importance discounting preserve past knowledge while freeing capacity for future tasks?
• RQ3 How does SLNID perform across varied datasets (permuted MNIST, CIFAR-100, Tiny ImageNet) and with different base LLL methods (MAS, EWC)?
• RQ4 What is the impact of SLNID on capacity usage (active/important neurons) and representations (sparsity/de-correlation) over sequential tasks?

Key findings

• Representation-based regularizers outperform parameter-based ones in sequential learning settings.
• The SLNID regularizer yields higher end-of-sequence accuracy and keeps more capacity available for future tasks across datasets.
• Local inhibition with neuron-importance discounting improves robustness to forgetting, achieving up to several percentage points gains over strong baselines on permuted MNIST, CIFAR-100, and Tiny ImageNet sequences.
• SLNID combined with MAS (and with EWC) consistently improves performance and allows smaller networks to match or exceed larger unregularized models.
• Ablation shows locality and importance discounting are crucial for performance, with SLNID producing sparser activations and leaving more unused parameters for future tasks.