[Paper Review] Progressive Neural Networks

Andrei A. Rusu, Neil C. Rabinowitz+6 more|arXiv (Cornell University)|Jun 15, 2016

Model Reduction and Neural Networks1,136 citations

TL;DR

Progressive neural networks add new columns for each task with lateral connections to previously learned features, enabling transfer without catastrophic forgetting in deep reinforcement learning across Atari, Pong variants, and 3D labyrinth tasks.

ABSTRACT

Learning to solve complex sequences of tasks--while both leveraging transfer and avoiding catastrophic forgetting--remains a key obstacle to achieving human-level intelligence. The progressive networks approach represents a step forward in this direction: they are immune to forgetting and can leverage prior knowledge via lateral connections to previously learned features. We evaluate this architecture extensively on a wide variety of reinforcement learning tasks (Atari and 3D maze games), and show that it outperforms common baselines based on pretraining and finetuning. Using a novel sensitivity measure, we demonstrate that transfer occurs at both low-level sensory and high-level control layers of the learned policy.

Motivation & Objective

• Advance continual learning by preventing catastrophic forgetting through a new task-specific column architecture.
• Enable transfer across sequential RL tasks via learned lateral connections to prior columns.
• Empirically evaluate transfer performance against finetuning baselines across diverse RL domains (Pong variants, Atari, Labyrinth).
• Develop analytical measures to understand where and how transfer occurs across task layers (Fisher information based).

Proposed method

• Instantiate a new network column for each task and freeze prior columns to avoid forgetting.
• Use lateral adapters to connect previous task features to the new column’s layers.
• Train with asynchronous A3C in multiple RL domains to assess transfer.
• Compare against baselines: single-task finetuning (top layer and full model) and two-column progressive nets.
• Analyze transfer using Average Fisher Sensitivity (AFS) and perturbation-based analysis.
• Address scalability and discuss potential pruning/compression of added capacity.

Experimental results

Research questions

• RQ1 Can progressive networks achieve positive transfer across heterogeneous RL tasks without forgetting prior tasks?
• RQ2 How does transfer manifest across low-level (visual) and high-level (policy) layers in sequential RL tasks?
• RQ3 Does adding more task columns yield diminishing capacity usage, suggesting feasible pruning or compression?
• RQ4 How does progressive transfer compare to standard finetuning baselines in RL domains like Pong variants, Atari, and Labyrinth?
• RQ5 What metrics (e.g., Fisher-based sensitivity) reveal where transfer occurs in the network?

Key findings

• Progressive networks frequently outperform strong finetuning baselines in transfer learning across RL tasks.
• Transfer can occur at both low-level sensory and high-level control layers, evidenced by analysis of feature reuse and AFS scores.
• Positive transfer increases with additional columns in several domains, though some negative transfer occurs when prior features dominate without new feature learning.
• A diminishing fraction of added capacity is actually utilized as more columns are added, suggesting pruning or compression could mitigate growth.
• The framework demonstrates transfer even between orthogonal or adversarial task pairs, e.g., Seaquest to Gopher in Atari, highlighting robust knowledge reuse.
• Forced adaptation (freezing prior columns) prevents forgetting and enables joint performance on multiple tasks without destructive interference.