[Paper Review] Predicting Deep Neural Network Generalization with Perturbation Response Curves

Yair Schiff, Brian Quanz+2 more|arXiv (Cornell University)|Jun 9, 2021

Neural Networks and Applications4 citations

TL;DR

This paper introduces a novel framework for predicting deep neural network generalization using Perturbation Response (PR) curves, which track accuracy changes under increasing training data perturbations. It proposes two new metrics—Gi-score and Pal-score—inspired by income inequality measures, achieving state-of-the-art performance in predicting generalization gaps across multiple tasks in the PGDL competition, while also enabling robust assessment of model invariance to data transformations.

ABSTRACT

The field of Deep Learning is rich with empirical evidence of human-like performance on a variety of prediction tasks. However, despite these successes, the recent Predicting Generalization in Deep Learning (PGDL) NeurIPS 2020 competition suggests that there is a need for more robust and efficient measures of network generalization. In this work, we propose a new framework for evaluating the generalization capabilities of trained networks. We use perturbation response (PR) curves that capture the accuracy change of a given network as a function of varying levels of training sample perturbation. From these PR curves, we derive novel statistics that capture generalization capability. Specifically, we introduce two new measures for accurately predicting generalization gaps: the Gi-score and Pal-score, which are inspired by the Gini coefficient and Palma ratio (measures of income inequality), that accurately predict generalization gaps. Using our framework applied to intra and inter-class sample mixup, we attain better predictive scores than the current state-of-the-art measures on a majority of tasks in the PGDL competition. In addition, we show that our framework and the proposed statistics can be used to capture to what extent a trained network is invariant to a given parametric input transformation, such as rotation or translation. Therefore, these generalization gap prediction statistics also provide a useful means for selecting optimal network architectures and hyperparameters that are invariant to a certain perturbation.

Motivation & Objective

• To address the lack of efficient, post-hoc measures for predicting deep neural network generalization gaps after training.
• To develop a framework that quantifies how robustly a trained model generalizes across varying levels of data perturbation.
• To enable the evaluation of model invariance to specific input transformations such as rotation, translation, and color jittering.
• To provide interpretable, robust statistics that outperform existing complexity measures in generalization gap prediction.
• To support architecture and hyperparameter selection by quantifying invariance to targeted perturbations.

Proposed method

• Construct Perturbation Response (PR) curves by evaluating model accuracy on training data perturbed at multiple magnitudes across different perturbation types (e.g., Mixup, rotation, translation).
• Define the Gi-score as a measure of the area between the PR curve and an ideal curve, reflecting sensitivity to perturbations.
• Define the Pal-score as the ratio of the top-performing perturbation level accuracy to the average accuracy, capturing peak robustness.
• Use the negative Pal-score plus Gi-score as a composite metric to rank models, with higher values indicating better generalization.
• Apply the framework to intra-class and inter-class Mixup, as well as geometric transformations like rotation and translation, to assess invariance.
• Train models on CIFAR-10, SVHN, and other benchmarks with varying augmentations and hyperparameters to evaluate predictive power.

Experimental results

Research questions

• RQ1 Can PR curves derived from multiple perturbation levels provide a more robust and informative measure of model generalization than single-point evaluations?
• RQ2 How well do the proposed Gi-score and Pal-score predict the generalization gap compared to existing state-of-the-art complexity measures?
• RQ3 To what extent can the framework quantify a model's invariance to specific parametric data transformations such as rotation or translation?
• RQ4 Can the framework guide the selection of optimal architectures and hyperparameters that enhance invariance to desired perturbations?
• RQ5 Does the composite Gi + (-Pal) score consistently outperform individual metrics and baselines across diverse datasets and model architectures?

Key findings

• The Gi-score and Pal-score combination achieved better generalization gap prediction scores than the current state-of-the-art on a majority of tasks in the PGDL competition.
• The framework outperformed baseline methods, including the winning PGDL submission, by capturing the full spectrum of perturbation responses rather than relying on a single interpolation magnitude.
• On the CINIC-10 dataset, the Gi-intraℓ0 & Pal-intraℓ0 combination achieved a composite score of 22.60, ranking highest among all metrics.
• The framework successfully captured model invariance to geometric transformations: for example, on CIFAR-10 with rotation, the Gi-intraℓ0 & Pal-intraℓ0 metric showed strong correlation with generalization gap (Pearson R > 0.8 in visualizations).
• The CMI score analysis confirmed that the proposed metrics provide distinct, informative predictive power beyond what is explained by hyperparameters alone.
• The method demonstrated robustness across diverse architectures (VGG, NiN, ResNet) and data types (CIFAR-10, SVHN, Fashion-MNIST), consistently outperforming baselines in generalization prediction.