[Paper Review] Continual Deep Learning by Functional Regularisation of Memorable Past

Pingbo Pan, Siddharth Swaroop+4 more|arXiv (Cornell University)|Apr 29, 2020

Domain Adaptation and Few-Shot LearningComputer Science31 references32 citations

TL;DR

FROMP introduces a functional-regularisation approach for continual learning by regularising network outputs on a small set of memorable past examples identified via a Gaussian-process formulation of DNNs, achieving state-of-the-art results on standard benchmarks.

ABSTRACT

Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. Recent works address this with weight regularisation. Functional regularisation, although computationally expensive, is expected to perform better, but rarely does so in practice. In this paper, we fix this issue by using a new functional-regularisation approach that utilises a few memorable past examples crucial to avoid forgetting. By using a Gaussian Process formulation of deep networks, our approach enables training in weight-space while identifying both the memorable past and a functional prior. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation and memory-based methods are naturally combined.

Motivation & Objective

• Address catastrophic forgetting in continual learning by focusing on output-level regularisation rather than solely weight regularisation.
• Introduce a Gaussian Process-based functional prior derived from a DNN to regularise the next task.
• Identify a small, informative set of memorable past examples to minimise computational overhead while preserving performance.
• Train in weight-space using a functional prior that exploits correlations among memorable past examples.

Proposed method

• Convert deep networks to Gaussian Processes (DNN2GP) to obtain a functional prior over network outputs.
• Identify memorable past examples by ranking data points using the per-example output Hessian/second-derivative (Lambda) as a relevance measure.
• Regularise the current task by minimising a loss that combines standard task loss with a closed-form functional-regularisation term derived from GP posteriors over memorable past outputs.
• Approximate the functional prior with manageable approximations (mean-field, block-diagonal kernels, deterministic optimization) to enable scalable training with Adam.
• Formulate an objective that resembles a weighted functional regulariser, encouraging outputs on memorable past to stay close to past predictions, while maintaining computational efficiency.

Experimental results

Research questions

• RQ1 Can functional regularisation over a small set of memorable past examples outperform traditional weight-regularisation in continual learning tasks?
• RQ2 Does leveraging a GP-based functional prior over network outputs improve retention of past tasks compared to existing functional-regularisation methods?
• RQ3 How does automatic selection of memorable past inputs influence performance and computational cost in continual learning?
• RQ4 What are the practical computational trade-offs when implementing FROMP with scalable approximations?

Key findings

• The FROMP approach yields state-of-the-art performance on standard continual learning benchmarks (per the abstract).
• Memorable past examples, chosen via a relevance measure tied to the loss Hessian, are often near decision boundaries and crucial for preventing forgetting.
• The GP-based functional prior uses correlations among memorable examples to regularise outputs without requiring full data retention, enabling training in weight-space.
• Compared to previous functional-regularisation methods, FROMP improves consistency and performance by incorporating a kernel over memorable past and regularising the mean toward past predictions.
• The method adapts standard optimisers (Adam) with modest computational overhead, thanks to approximations such as block-diagonal kernels and mean-field assumptions.