[Paper Review] Sequential mastery of multiple visual tasks: Networks naturally learn to learn and forget to forget

Guy Davidson, Michael C. Mozer|arXiv (Cornell University)|May 26, 2019

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper demonstrates that standard convolutional neural networks naturally exhibit forward facilitation and reduced backward interference when sequentially learning multiple visual tasks, without specialized architectures or regularization. As networks master more tasks, they learn new ones faster and forget older ones less—suggesting that broader domain exposure inherently improves continual learning, mirroring human-like learning scalability.

ABSTRACT

We explore the behavior of a standard convolutional neural net in a continual-learning setting that introduces visual classification tasks sequentially and requires the net to master new tasks while preserving mastery of previously learned tasks. This setting corresponds to that which human learners face as they acquire domain expertise serially, for example, as an individual studies a textbook. Through simulations involving sequences of ten related visual tasks, we find reason for optimism that nets will scale well as they advance from having a single skill to becoming multi-skill domain experts. We observe two key phenomena. First, \\emph{forward facilitation}---the accelerated learning of task $n+1$ having learned $n$ previous tasks---grows with $n$. Second, \\emph{backward interference}---the forgetting of the $n$ previous tasks when learning task $n+1$---diminishes with $n$. Amplifying forward facilitation is the goal of research on metalearning, and attenuating backward interference is the goal of research on catastrophic forgetting. We find that both of these goals are attained simply through broader exposure to a domain.

Motivation & Objective

• To investigate how standard neural networks scale in continual learning across multiple sequential visual tasks.
• To examine whether forward facilitation (faster learning of new tasks) and reduced backward interference (less forgetting of old tasks) emerge naturally with increasing task exposure.
• To assess whether broader domain experience alone can mitigate catastrophic forgetting and enhance metalearning-like benefits, without specialized mechanisms.
• To draw parallels between neural network learning dynamics and human cognitive learning patterns, particularly distributed practice and task similarity effects.

Proposed method

• Training a standard convolutional neural network on a sequence of ten related visual classification tasks in a strictly sequential order.
• Measuring training trials required to reach criterion performance on each new task while maintaining performance on previously learned tasks.
• Quantifying backward interference by measuring performance drop on prior tasks after learning a new task, and tracking relearning effort to restore performance.
• Analyzing scaling behavior across tasks, including power-law learning curves and nonmonotonic patterns in interference and facilitation.
• Testing task similarity effects by varying task dimensions (e.g., texture vs. shape) and introducing task-based modulation of visual processing.
• Using log-log plots to assess power-law scaling of learning efficiency with respect to number of tasks and training exposure.

Experimental results

Research questions

• RQ1 Does forward facilitation—faster learning of later tasks—emerge naturally as a network acquires more visual tasks?
• RQ2 Does backward interference—forgetting of previously learned tasks—decrease as the network gains more domain expertise through sequential learning?
• RQ3 How does the number of previously learned tasks affect the learning efficiency and stability of new task acquisition?
• RQ4 To what extent does task similarity influence interference and facilitation in sequential continual learning?
• RQ5 Can distributed, interleaved practice through sequential task exposure reduce forgetting without requiring specialized regularization or replay mechanisms?

Key findings

• Forward facilitation grows with the number of previously learned tasks: networks learn new tasks faster as they accumulate domain expertise, indicating emergent metalearning capabilities.
• Backward interference diminishes with increasing task count: performance on earlier tasks degrades less when new tasks are introduced, suggesting improved memory stability over time.
• The relearning effort required to restore performance on prior tasks decreases significantly with repeated exposure, indicating durable memory formation through distributed practice.
• Learning efficiency follows a power-law scaling with respect to both the number of tasks learned and the amount of training per task, a hallmark of human skill acquisition.
• The anomalous increase in forgetting for task 2 suggests a transitional phase, possibly analogous to the 'zero-one-infinity' principle, where early task pairs show higher interference.
• Reducing task similarity (e.g., by varying task dimensions) and introducing task-based modulation of visual processing both mitigate interference, supporting the role of perceptual distinctiveness in reducing interference.