[Paper Review] In Search of Robust Measures of Generalization

Gintare Karolina Dziugaite, Alexandre Drouin+6 more|arXiv (Cornell University)|Oct 22, 2020

Domain Adaptation and Few-Shot Learning27 references4 citations

TL;DR

This paper proposes a distributional robustness framework to evaluate generalization measures in deep learning, arguing that average-case performance in prior studies (e.g., [9]) can obscure critical failures. By focusing on worst-case environments across data distributions, architectures, and hyperparameters, the authors identify that path-norm and PAC-Bayes flatness measures are more robust, while spectral measures consistently fail, especially under width and depth variations.

ABSTRACT

One of the principal scientific challenges in deep learning is explaining generalization, i.e., why the particular way the community now trains networks to achieve small training error also leads to small error on held-out data from the same population. It is widely appreciated that some worst-case theories -- such as those based on the VC dimension of the class of predictors induced by modern neural network architectures -- are unable to explain empirical performance. A large volume of work aims to close this gap, primarily by developing bounds on generalization error, optimization error, and excess risk. When evaluated empirically, however, most of these bounds are numerically vacuous. Focusing on generalization bounds, this work addresses the question of how to evaluate such bounds empirically. Jiang et al. (2020) recently described a large-scale empirical study aimed at uncovering potential causal relationships between bounds/measures and generalization. Building on their study, we highlight where their proposed methods can obscure failures and successes of generalization measures in explaining generalization. We argue that generalization measures should instead be evaluated within the framework of distributional robustness.

Motivation & Objective

• To address the lack of systematic, robust evaluation methods for generalization bounds in deep learning.
• To critique the average-case evaluation framework used in prior large-scale studies, such as [9], which may mask failures of generalization measures.
• To propose a distributional robustness framework that identifies failures in specific environments rather than averaging over them.
• To improve empirical evaluation of generalization theories by emphasizing worst-case performance across diverse model and data configurations.
• To guide future development of more reliable, theory-driven measures of generalization through robust regression and targeted interventions.

Proposed method

• Adopt a distributional robustness framework where a generalization measure's strength is judged by its worst-case performance across a family of environments.
• Define families of environments based on data distributions (CIFAR-10, SVHN), model architectures, training set sizes, and hyperparameters (depth, width, learning rate).
• Use robust regression techniques (from Appendix B) to model the relationship between hyperparameters and generalization gaps, focusing on worst-case deviations.
• Apply filtering to remove environments with high Monte Carlo noise, ensuring reliable sign-error estimation.
• Compare generalization measures (e.g., path-norm, spectral norms, PAC-Bayes flatness) across subfamilies of environments to isolate failure modes.
• Use sign-error analysis to assess whether measures predict generalization gaps better than random chance, with emphasis on worst-case subfamilies.

Experimental results

Research questions

• RQ1 Can average-case performance metrics in prior studies obscure critical failures of generalization measures across diverse model and data configurations?
• RQ2 How do different generalization measures perform under worst-case conditions across families of environments defined by architecture, data, and hyperparameters?
• RQ3 Which generalization measures exhibit robustness across diverse interventions such as changes in network depth, width, and learning rate?
• RQ4 To what extent do existing theoretical bounds fail to capture the correct dependency of generalization error on training set size?
• RQ5 Can a robust evaluation framework based on worst-case performance improve the identification of reliable generalization measures?

Key findings

• The average-case performance of generalization measures in [9] can obscure critical failures, as shown by the fact that 8.2% of width environments were discarded due to high Monte Carlo noise.
• Path-norm and PAC-Bayes flatness-based measures consistently outperform others in worst-case environments, particularly in 'Depth' and 'Width' subfamilies for CIFAR-10.
• Spectral measures such as 'prod.of.spec' perform poorly across all environments, especially under width and depth variations, indicating a fundamental lack of robustness.
• The study identifies that many existing bounds grow with training set size, contradicting the empirical observation that generalization error shrinks with more data.
• The proposed robust evaluation framework reveals that no measure performs significantly better than random chance in 'Depth' environments for CIFAR-10 when evaluated under worst-case conditions.
• The authors find that differences in hyperparameter ranges and the use of dropout in [9] lead to divergent conclusions, highlighting the sensitivity of results to experimental design.