[Paper Review] An Empirical Study of Example Forgetting during Deep Neural Network Learning

Mariya Toneva, Alessandro Sordoni+4 more|arXiv (Cornell University)|Dec 12, 2018

Domain Adaptation and Few-Shot LearningComputer Science37 references199 citations

TL;DR

This paper defines and analyzes forgetting events for individual training examples during SGD, finding unforgettable and forgettable examples, cross-architecture stability, and that removing forgettable examples often preserves generalization.

ABSTRACT

Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.

Motivation & Objective

• Investigate whether a forgetting phenomenon occurs within a single-task learning process similar to catastrophic forgetting.
• Characterize the distribution and properties of forgetting events across datasets and architectures.
• Assess whether removing forgettable or unforgettable examples impacts generalization and data efficiency.

Proposed method

• Define forgetting events as moments when an example transitions from correct to incorrect during SGD training.
• Compute per-example forgetting statistics as training progresses, using mini-batch updates.
• Evaluate across MNIST, permuted MNIST, and CIFAR-10 with CNN, ResNet, and WideResNet architectures.
• Analyze correlation between forgetting events and misclassification margin.
• Experiment with removing subsets of data ordered by forgetting events to test data efficiency.

Experimental results

Research questions

• RQ1 Do neural networks exhibit forgetting events for individual training examples within a single task?
• RQ2 Are some examples unforgettable across seeds and architectures, and do forgetting patterns generalize across models?
• RQ3 Can forgetting dynamics identify informative vs noisy or outlier examples, and how does removing such examples affect generalization?

Key findings

• Many examples are unforgettable, stable across seeds, and correlate across architectures.
• The most forgettable examples often have noisy labels or uncommon features and are visually ambiguous.
• Removing a large fraction of the least-forgotten examples does not harm generalization; removing the most-forgotten examples degrades less when choosing data strategically.
• For CIFAR-10, up to 30-35% of data can be removed based on forgetting without significant performance loss.
• Forgetful examples tend to lie near the decision boundary, behaving like data points similar to SVM support vectors.
• Forgetting statistics remain stable across epochs and architectures, enabling transfer of forgetting orderings between models.