[Paper Review] What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation

Vitaly Feldman, Chiyuan Zhang|arXiv (Cornell University)|Aug 9, 2020

Anomaly Detection Techniques and ApplicationsComputer Science32 references95 citations

TL;DR

The paper empirical validates the long tail memorization theory by estimating memorization and influence of training examples, showing memorized instances significantly aid generalization and many high-influence pairs arise from single training examples. It also analyzes cross-architecture consistency and the depth where memorization occurs.

ABSTRACT

Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given. In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).

Motivation & Objective

• Motivate and test the long tail theory that memorization aids generalization in long-tailed data distributions.
• Develop efficient estimators for memorization and influence that are feasible for large datasets.
• Quantify the marginal utility of memorized examples and identify high-influence train-test pairs.
• Assess how memorization and influence vary across architectures and data regimes.
• Investigate where memorization primarily resides within the network representations.

Proposed method

• Define memorization as the change in h(x_i) = y_i probability when the i-th example is added to the training set (Eq. 1).
• Introduce a subsampled estimator mem_m for memorization using random subsets of size m to enable feasible estimation.
• Define a subsampled influence infl_m that estimates the effect of including a training example on test accuracy, averaged over random subsets.
• Train models on random subsets of size m and computePr(h_k(x) = y) conditioned on i in or out of the subset to estimate memorization and influence.
• Provide selection thresholds theta_mem = 0.25 and theta_infl = 0.15 to identify high-memorization and high-influence pairs.
• Experiment with ResNet50 on ImageNet, CIFAR-100, and MNIST to estimate memorization and influence, and compare marginal utility versus random subset baselines.
• Examine consistency across architectures and the claim that memorization largely resides in deep representations rather than the last layer.

Experimental results

Research questions

• RQ1 Does memorization meaningfully contribute to generalization in long-tailed data distributions as proposed by Fel19?
• RQ2 Can efficient subsampling yield accurate memorization and influence estimates for large datasets?
• RQ3 Do memorized examples have higher marginal utility than randomly chosen examples for improving test accuracy?
• RQ4 Are high-influence train-test pairs concentrated around single training examples and are they interpretable visually?
• RQ5 Where in the network does memorization primarily occur (last layer vs representation) across architectures?

Key findings

• A substantial fraction of memorized examples exist in CIFAR-100 and ImageNet, with some memorization estimates ≥ 0.3 and notable marginal utility when removed.
• Memorized examples have higher marginal utility than equally sized random subsets, indicating memorization contributes beyond merely reducing sample count.
• A significant number of high-influence train-test pairs exist (e.g., 1641 pairs in ImageNet) and many test examples are influenced by a single training example (1298 test examples).
• High-influence pairs are often visually interpretable and reflect meaningful similarity or near-duplicates, highlighting subpopulations in the long tail.
• Removing memorized examples reduces test accuracy; for CIFAR-100, the accuracy drop when removing memorized sets is larger for high-influence portions (2.38% contribution) than overall.
• Most memorization arises in the deep representation rather than the final layer, as shown by the limited effectiveness of training only a linear classifier on fixed representations.