[Paper Review] Reinforced Continual Learning

Ju Xu, Zhanxing Zhu|arXiv (Cornell University)|May 31, 2018

Domain Adaptation and Few-Shot LearningComputer Science14 references152 citations

TL;DR

The paper introduces Reinforced Continual Learning (RCL), which uses reinforcement learning to adaptively expand neural networks for each new task, aiming to maximize validation accuracy while minimizing model complexity and forgetting. RCL outperforms several baselines on sequential MNIST variants and Incremental CIFAR-100 with fewer added parameters.

ABSTRACT

Most artificial intelligence models have limiting ability to solve new tasks faster, without forgetting previously acquired knowledge. The recently emerging paradigm of continual learning aims to solve this issue, in which the model learns various tasks in a sequential fashion. In this work, a novel approach for continual learning is proposed, which searches for the best neural architecture for each coming task via sophisticatedly designed reinforcement learning strategies. We name it as Reinforced Continual Learning. Our method not only has good performance on preventing catastrophic forgetting but also fits new tasks well. The experiments on sequential classification tasks for variants of MNIST and CIFAR-100 datasets demonstrate that the proposed approach outperforms existing continual learning alternatives for deep networks.

Motivation & Objective

• Motivate continual learning that scales by adaptively expanding network capacity as tasks arrive.
• Prevent forgetting by freezing previous task parameters while training added parameters.
• Leverage reinforcement learning to automatically search for near-optimal architectural expansions per task.

Proposed method

• Controller (LSTM) generates a sequence of actions determining how many filters/nodes to add per layer for a new task.
• Task network expands adaptively; only newly added parameters are trained for the new task to avoid semantic drift.
• Reward combines validation accuracy and network complexity to balance performance and efficiency (R_t = A_t + α C_t).
• Policy gradient with an actor-critic framework updates the controller and a value network to maximize expected reward.
• Training proceeds task-by-task, expanding the network according to controller outputs and freezing previous task parameters.

Experimental results

Research questions

• RQ1 Can reinforcement learning effectively search for optimal architectural expansions to mitigate forgetting in continual learning?
• RQ2 How does adaptive expansion influence accuracy versus model complexity across sequential tasks?
• RQ3 Does RCL prevent forgetting better than fixed-size or other expandable architectures while using fewer added parameters?

Key findings

• RCL achieves better accuracy and smaller model size than PGN and DEN across MNIST permutations, MNIST mix, and Incremental CIFAR-100.
• RCL reduces the number of added parameters by substantial margins (e.g., 42% and 53% reductions for CIFAR-100 compared to PGN and DEN).
• Forgetting is mitigated in RCL and PGN, while fixed-size approaches exhibit catastrophic forgetting. DEN does not fully prevent forgetting due to retraining previous parameters.
• Increasing α (weight on model complexity) reduces parameters but can slightly degrade accuracy, enabling a tradeoff between performance and size.
• RCL uses fewer hyperparameters than DEN and demonstrates more stable performance across settings.