[Paper Review] Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Xilai Li, Yingbo Zhou+3 more|arXiv (Cornell University)|Mar 30, 2019

Multimodal Machine Learning ApplicationsComputer Science110 citations

TL;DR

The paper proposes Learn-to-Grow, a continual learning framework that explicitly separates structure learning from parameter learning, using neural architecture search to grow task-specific structures while reusing shared primitives to mitigate catastrophic forgetting. It demonstrates strong performance gains on permuted MNIST, split CIFAR-100, and Visual Domain Decathlon.

ABSTRACT

Addressing catastrophic forgetting is one of the key challenges in continual learning where machine learning systems are trained with sequential or streaming tasks. Despite recent remarkable progress in state-of-the-art deep learning, deep neural networks (DNNs) are still plagued with the catastrophic forgetting problem. This paper presents a conceptually simple yet general and effective framework for handling catastrophic forgetting in continual learning with DNNs. The proposed method consists of two components: a neural structure optimization component and a parameter learning and/or fine-tuning component. By separating the explicit neural structure learning and the parameter estimation, not only is the proposed method capable of evolving neural structures in an intuitively meaningful way, but also shows strong capabilities of alleviating catastrophic forgetting in experiments. Furthermore, the proposed method outperforms all other baselines on the permuted MNIST dataset, the split CIFAR100 dataset and the Visual Domain Decathlon dataset in continual learning setting.

Motivation & Objective

• Motivate continual learning as a means to prevent catastrophic forgetting in sequential task learning.
• Propose a two-component framework that separates structure growth from parameter learning.
• Develop a differentiable NAS-based method to decide when to reuse, adapt, or create new components for each task.
• Demonstrate improved performance and reduced forgetting on diverse continual learning benchmarks.

Proposed method

• Introduce a Learn-to-Grow framework that expands the model Tt = Tt-1 ∪ θt with task-specific structure st(Θt).
• Formulate a penalized loss with structure regularization Rs(s t) and parameter regularization Rp(Θt) to bound growth.
• Use differentiable neural architecture search (NAS) to optimize the per-layer choices: reuse, adaptation, or new, via a continuous relaxation of architecture weights α.
• Adopt a two-stage optimization: NAS to select architecture using a validation set and then parameter learning on a training set; update a super network that aggregates all task-specific structures.
• Implement three concrete operations for each layer: reuse (share previous weights), adaptation (add a small adapter), and new (spawn a fresh layer).
• Retrain the selected structure for the current task, with optional fine-tuning of reused parts under regularization to balance stability and plasticity.

Experimental results

Research questions

• RQ1 Can explicit continual structure learning via NAS produce sensible, task-tailored architectures for sequential tasks?
• RQ2 Does separating structure learning from parameter learning reduce catastrophic forgetting more effectively than baseline continual learning methods?
• RQ3 How does the Learn-to-Grow framework perform on standard continual learning benchmarks (permuted MNIST, Visual Domain Decathlon) compared to state-of-the-art methods?
• RQ4 What is the impact of different architectural decisions (reuse/adaptation/new) on forgetting and transfer across tasks.

Key findings

• The learned structures tend to reuse layers for semantically similar tasks and spawn new ones when tasks are very different, producing sensible task-specific architectures.
• Forgetting is significantly mitigated compared to SGD baseline and competitive with or better than other methods on permuted MNIST and Visual Domain Decathlon.
• The approach achieves strong average accuracy across tasks and favorable parameter efficiency, often outperforming baselines with similar model sizes.
• On permuted MNIST, the method reaches higher average accuracy than several baselines including EWC, IMM, HAT, and DEN, depending on regularization.
• On the Visual Domain Decathlon, the Learn-to-Grow method achieves the best overall results across multiple tasks with a competitive parameter footprint.
• The method demonstrates that explicit structure learning reduces the need for extensive fine-tuning while maintaining or improving performance across sequential tasks.