[Paper Review] Detecting Memorization in ReLU Networks

Edo Collins, Siavash Bigdeli+1 more|arXiv (Cornell University)|Oct 8, 2018

Neural Networks and Reservoir Computing25 references4 citations

TL;DR

This paper introduces a novel method to detect memorization in ReLU neural networks by measuring the non-linearity of deep layer activations using non-negative matrix factorization (NMF) on single-class input batches. The approach reveals that high non-linearity in deep layers correlates strongly with memorization, and the method enables effective early stopping by tracking the area under the curve (AuC) of NMF compression performance.

ABSTRACT

We propose a new notion of `non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization. Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases. We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets. Our results demonstrate that as an indicator for memorization, our technique can be used to perform early stopping.

Motivation & Objective

• To identify a reliable, activation-based indicator of memorization in deep ReLU networks that is robust to reversible transformations like weight scaling.
• To investigate whether non-linearity in deep layers—measured via non-negative matrix factorization (NMF) of activation matrices—correlates with poor generalization.
• To evaluate whether the proposed method can serve as a practical early stopping criterion during training to prevent overfitting.
• To compare the sensitivity of NMF against PCA and random ablation as compression-based robustness measures for detecting memorization.

Proposed method

• Define non-linearity of a ReLU layer with respect to an input batch as its proximity to a linear system, quantified by the non-negative rank of the activation matrix.
• Apply approximate non-negative matrix factorization (NMF) to activation matrices from single-class batches to estimate non-linearity across a grid of approximation ranks $k$.
• Measure the area under the curve (AuC) of accuracy vs. $k$ for NMF-compressed activations to assess robustness and detect memorization.
• Compare NMF with PCA and random ablation techniques by computing their respective AuC curves and evaluating correlation with generalization error.
• Perform layer-by-layer analysis to trace the progression of non-linearity through the network during training.
• Use the AuC of single-class NMF as a real-time signal to trigger early stopping when memorization begins.

Experimental results

Research questions

• RQ1 Can non-linearity in deep ReLU layers, as measured by non-negative rank of activation matrices, serve as a reliable indicator of memorization?
• RQ2 How does the non-linearity of activations differ between single-class and multi-class input batches, and what does this reveal about memorization?
• RQ3 Does the NMF-based non-linearity measure correlate more strongly with generalization error than PCA or random ablation?
• RQ4 Can the NMF-based AuC curve be used to detect the onset of memorization during training and enable effective early stopping?

Key findings

• High non-linearity in deep layers, as measured by NMF on single-class batches, is a strong indicator of memorization, especially when compared to multi-class batches.
• The NMF-based AuC curve shows a Pearson correlation of -0.82 with generalization error on CIFAR-10, outperforming PCA (-0.64) and random ablation (-0.61).
• The peak of the NMF AuC curve aligns almost exactly with the local minimum of test loss, indicating that the method detects the onset of overfitting with high precision.
• The method successfully identifies memorization in both fully-connected and convolutional networks trained on image and audio datasets.
• Layer-by-layer analysis reveals that memorization occurs in distinct phases, with non-linearity increasing significantly in deeper layers as training progresses.
• The proposed NMF-based approach enables effective early stopping, as the AuC curve reaches its maximum just before test error begins to rise.