[Paper Review] Gradient based sample selection for online continual learning

Rahaf Aljundi, Min Lin+2 more|arXiv (Cornell University)|Mar 20, 2019

Domain Adaptation and Few-Shot LearningComputer Science26 references79 citations

TL;DR

The paper formulates replay-buffer population as a constraint reduction problem and introduces a gradient-based surrogate to maximize diversity of stored samples, enabling online continual learning without task boundaries. It provides both an exact (IQP) and a cheap greedy method, showing competitive results across benchmarks.

ABSTRACT

A continual learning agent learns online with a non-stationary and never-ending stream of data. The key to such learning process is to overcome the catastrophic forgetting of previously seen data, which is a well known problem of neural networks. To prevent forgetting, a replay buffer is usually employed to store the previous data for the purpose of rehearsal. Previous works often depend on task boundary and i.i.d. assumptions to properly select samples for the replay buffer. In this work, we formulate sample selection as a constraint reduction problem based on the constrained optimization view of continual learning. The goal is to select a fixed subset of constraints that best approximate the feasible region defined by the original constraints. We show that it is equivalent to maximizing the diversity of samples in the replay buffer with parameters gradient as the feature. We further develop a greedy alternative that is cheap and efficient. The advantage of the proposed method is demonstrated by comparing to other alternatives under the continual learning setting. Further comparisons are made against state of the art methods that rely on task boundaries which show comparable or even better results for our method.

Motivation & Objective

• Motivate continual learning under online, non-i.i.d. data streams with no task boundaries.
• Formulate replay buffer population as a constraint reduction problem.
• Introduce a surrogate objective based on gradient diversity to approximate feasible-region minimization.
• Provide efficient online algorithms (IQP and greedy) for buffer population.
• Demonstrate competitive performance on benchmark continual learning datasets without task boundary assumptions.

Proposed method

• Formulate continual learning as a constrained optimization problem over model parameters with constraints from past data.
• Express the feasible region via gradient inner products and aim to preserve it using a fixed-size replay buffer.
• Propose a surrogate objective that minimizes the sum of normalized gradient inner products to maximize diversity (Eq. 7).
• Relate the surrogate to solid-angle minimization and show equivalence to maximizing gradient direction variance (Eq. 8).
• Offer an exact gradient-based IQP method (Algorithm 1) for selecting a subset of constraints (samples) to minimize the surrogate objective.
• Present a cheaper greedy alternative (Algorithm 2) that scores samples by maximal cosine similarity to a random subset and probabilistically replaces buffer items.
• Discuss how rehearsal (regularization) relates to constrained optimization and empirically compare approaches.

Experimental results

Research questions

• RQ1 Can replay buffers be populated effectively without knowledge of task boundaries or i.i.d. assumptions?
• RQ2 Does a gradient-based diversity criterion reliably approximate the feasible-region minimization implied by the original constraints?
• RQ3 Are online gradient-based selection methods (IQP and greedy) computationally viable and empirically competitive across standard continual learning benchmarks?
• RQ4 How do the proposed methods compare to reservoir sampling and task-aware replay baselines under non-stationary data streams?

Key findings

• The gradient-based surrogate correlates monotonically with the solid-angle measure of the feasible region, justifying its use as an objective.
• The online gradient-based methods outperform random sampling and several clustering-based baselines across MNIST and CIFAR-10 tasks under non-task-boundary settings.
• The greedy variant (GSS-Greedy) is computationally efficient and performs on par with or better than other strategies, especially on CIFAR-10.
• When compared to reservoir sampling on imbalanced data sequences, the proposed methods achieve higher average accuracy and robustness to under-represented tasks.
• The methods achieve performance similar to or better than some task-aware replay baselines (e.g., GEM, iCaRL) despite not using task boundary information.