[Paper Review] Fantastic Generalization Measures and Where to Find Them

Yiding Jiang, Behnam Neyshabur+3 more|arXiv (Cornell University)|Dec 4, 2019

Machine Learning and Algorithms41 references153 citations

TL;DR

A large-scale empirical study evaluating 40 complexity/ generalization measures across over 10,000 CNNs trained with varied hyperparameters to identify which measures reliably relate to generalization.

ABSTRACT

Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization in deep networks. We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies. We train over 10,000 convolutional networks by systematically varying commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.

Motivation & Objective

• Assess how well established and empirical complexity measures predict generalization across diverse models.
• Quantify reliability of measures using extensive controlled experiments to avoid spurious correlations.
• Investigate causal-ish relationships between measures and generalization with novel evaluation tools.
• Provide guidance on promising directions for future theory and practice in generalization measurement.

Proposed method

• Train 2187 CIFAR-10 models by varying 7 hyperparameters with 3 choices each until convergence criteria are met.
• Compute 40 complexity measures spanning PAC-Bayes, VC-dimension, norm-based, and empirical measures for each model.
• Evaluate measure-generalization alignment using Kendall’s rank correlation tau across all models.
• Introduce granulated correlation via per-hyperparameter analysis to detect parameter-specific predictive power (Psi).
• Employ a conditional independence framework inspired by the IC algorithm to assess potential causal links between measures and generalization.
• Repeat analyses on SVHN to test robustness of findings across datasets.

Experimental results

Research questions

• RQ1 Which complexity measures show robust correlation with generalization across a wide hyperparameter space?
• RQ2 Do norm-based or margin/sharpness-related measures reliably predict generalization, and under what training conditions?
• RQ3 Can we distinguish spurious correlations from causal relationships between measures and generalization using controlled experiments and conditional independence?
• RQ4 How do different hyperparameters influence the predictive power of various measures?
• RQ5 Are sharpness-based and optimization-related measures more predictive of generalization than traditional bounds?

Key findings

• Some complexity measures exhibit spurious correlations and can be misleading without careful experimental design.
• Many norm-based measures correlate negatively with generalization when optimization injects stochasticity, notably the product of spectral norms bound.
• Sharpness-based measures, including PAC-Bayes style bounds, perform best overall as predictors of generalization gap.
• Optimization-related measures, such as gradient noise and optimization speed, show predictive power for generalization.
• Spectral complexity behaves unexpectedly; spectral bound correlates negatively with generalization, while Frobenius distance to initialization can be negatively correlated, and Frobenius parameter norm can be positively correlated.
• Empirical measures like cross-entropy, margin, and output entropy align with general understanding that larger margins and higher output entropy relate to better generalization.