[Paper Review] Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach

Wenda Zhou, Victor Veitch+3 more|arXiv (Cornell University)|Apr 16, 2018

Domain Adaptation and Few-Shot LearningComputer Science23 references81 citations

TL;DR

The paper derives non-vacuous PAC-Bayesian generalization bounds for realistic neural networks by linking compression size to generalization, achieving the first non-vacuous bounds on ImageNet-scale models using off-the-shelf compression methods.

ABSTRACT

Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be "compressed" to much smaller representations. The purpose of this paper is to connect these two empirical observations. Our main technical result is a generalization bound for compressed networks based on the compressed size. Combined with off-the-shelf compression algorithms, the bound leads to state of the art generalization guarantees; in particular, we provide the first non-vacuous generalization guarantees for realistic architectures applied to the ImageNet classification problem. As additional evidence connecting compression and generalization, we show that compressibility of models that tend to overfit is limited: We establish an absolute limit on expected compressibility as a function of expected generalization error, where the expectations are over the random choice of training examples. The bounds are complemented by empirical results that show an increase in overfitting implies an increase in the number of bits required to describe a trained network.

Motivation & Objective

• Motivate the link between neural network compressibility and generalization in deep learning.
• Develop a PAC-Bayes bound that incorporates explicit compressed model size as a complexity term.
• Show that using standard compression schemes yields non-vacuous generalization guarantees on practical architectures.
• Demonstrate that overfitting imposes limits on compressibility and has empirical support.
• Provide practical guidance for applying the bound to real networks using existing compression tools.

Proposed method

• Embed compression-derived code length into a PAC-Bayes bound to relate generalization to effective compressed size.
• Use a point-mass PAC-Bayes posterior at the compressed model and compute KL divergence with a computable prior.
• Extend the bound to account for robustness to weight perturbations via a structured compression output (S, C, Q) with a stochastic Gaussian perturbation over nonzeros.
• Derive a bound (Theorem 4.3) that includes both the encoded size and a perturbation-robustness term.
• Apply the bound to experiments with MNIST (LeNet-5) and ImageNet-scale networks using pruning and quantization (and noise augmentation in forward passes).
• Argue that the bound is compatible with various compression schemes and does not require retraining for tightness.

Experimental results

Research questions

• RQ1 Can a PAC-Bayesian bound be tightened by explicitly accounting for the compressed size of a trained network?
• RQ2 Do standard compression techniques yield non-vacuous generalization bounds on modern architectures, including ImageNet-scale models?
• RQ3 Is there a detectable relationship between overfitting and the compressibility of neural networks in practice?
• RQ4 How does robustness to weight perturbations interact with compression-based generalization bounds?

Key findings

• The bound based on compressed size yields state-of-the-art, non-vacuous generalization guarantees for realistic networks.
• Non-vacuous bounds are achieved for ImageNet-scale models by combining compression with PAC-Bayesian theory.
• There is empirical evidence that increased overfitting worsens compressibility under a fixed compression scheme.
• Compressibility of highly overfitting models is limited, and robustness to weight perturbations can be incorporated to improve the bound.
• Experiments on MNIST and ImageNet-scale networks show effective compressed sizes that lead to meaningful error bounds.