[Paper Review] Lifelong Learning with Dynamically Expandable Networks

Jaehong Yoon, Eunho Yang+2 more|arXiv (Cornell University)|Aug 4, 2017

Domain Adaptation and Few-Shot LearningComputer Science14 references181 citations

TL;DR

DEN is a deep network that dynamically expands and selectively retrains to learn a sequence of tasks, achieving comparable or better performance than batch-trained models with far fewer parameters and without semantic drift.

ABSTRACT

We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets under lifelong learning scenarios, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch counterparts with substantially fewer number of parameters. Further, the obtained network fine-tuned on all tasks obtained significantly better performance over the batch models, which shows that it can be used to estimate the optimal network structure even when all tasks are available in the first place.

Motivation & Objective

• Motivate and address lifelong learning for deep networks where tasks arrive sequentially.
• Develop a model that selectively retrains existing components and expands capacity only when needed.
• Prevent semantic drift (catastrophic forgetting) via timestamped units and adaptive splitting.
• Achieve competitive task performance with fewer parameters compared to batch or static lifelong-learning baselines.
• Demonstrate that the model can estimate optimal network structure even when all tasks are available later.]
• -

Proposed method

• Introduce Dynamically Expandable Networks (DEN) that fuse selective retraining, dynamic expansion, and split/duplication of units.
• Use sparse and group-sparse regularization to identify relevant subnetworks and to decide how many neurons to add per layer (Eq. 5).
• Apply selective retraining by solving targeted optimization (Eq. 3 and Eq. 4) within a subset S of the network.
• Expand capacity only when needed by solving a group-sparsity objective that can prune unnecessary added units (Eq. 5).
• Split/duplicate units whose semantic drift (measured by drift rho_i^t) exceeds a threshold, then retrain (Eq. 6 and Algorithm 4).
• Timestamp newly added units to enable inference that respects training stage (timestamped inference).

Experimental results

Research questions

• RQ1 Can a neural network learn a sequence of tasks efficiently by selectively retraining only relevant parts of the network?
• RQ2 Does dynamic capacity expansion improve performance when new tasks are semantically different from previous ones?
• RQ3 Can splitting/duplicating units and timestamped inference prevent semantic drift while enabling continual learning?
• RQ4 How does DEN compare to baselines (e.g., Elastic Weight Consolidation, Progressive Networks) in terms of accuracy and parameter efficiency?
• RQ5 Can the learned structure generalize to estimate optimal capacity when all tasks are available (batch setting)?

Key findings

• DEN achieves similar or better performance than batch models while using 11.9%–60.3% of the parameters across datasets.
• DEN outperforms baselines like L2-regularized retraining, EWC, and Progressive Networks on MNIST-Variation, CIFAR-100, and AWA in lifelong learning scenarios.
• Selective retraining reduces training time and focuses updates on task-relevant subnetworks.
• Dynamic expansion with group sparsity adds only the necessary neurons, avoiding over-parameterization and improving efficiency.
• Subnet/feature drift triggers unit duplication, helping preserve older task performance while acquiring new knowledge.
• Fine-tuning DEN on all tasks (DEN-Finetune) yields the best performance, surpassing batch models, and enabling effective network capacity estimation.