[Paper Review] Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience

Vaishnavh Nagarajan, J. Zico Kolter|arXiv (Cornell University)|May 30, 2019

Advanced Neural Network ApplicationsComputer Science45 citations

TL;DR

This paper introduces a general PAC-Bayesian framework that leverages noise-resilience of deep networks to derive a generalization bound for the original deterministic, uncompressed network, applied to deep ReLU nets and avoiding depth blowup from spectral-norm products.

ABSTRACT

The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) finds solutions that lie in flat, wide minima in the training loss -- minima where the output of the network is resilient to small random noise added to its parameters. So far this observation has been used to provide generalization guarantees only for neural networks whose parameters are either extit{stochastic} or extit{compressed}. In this work, we present a general PAC-Bayesian framework that leverages this observation to provide a bound on the original network learned -- a network that is deterministic and uncompressed. What enables us to do this is a key novelty in our approach: our framework allows us to show that if on training data, the interactions between the weight matrices satisfy certain conditions that imply a wide training loss minimum, these conditions themselves {\em generalize} to the interactions between the matrices on test data, thereby implying a wide test loss minimum. We then apply our general framework in a setup where we assume that the pre-activation values of the network are not too small (although we assume this only on the training data). In this setup, we provide a generalization guarantee for the original (deterministic, uncompressed) network, that does not scale with product of the spectral norms of the weight matrices -- a guarantee that would not have been possible with prior approaches.

Motivation & Objective

• Understand why overparameterized deep nets generalize well and how SGD finds wide, noise-resilient minima.
• Develop a PAC-Bayesian framework that yields generalization bounds for deterministic, uncompressed networks using training-time noise-resilience.
• Specialize the framework to deep ReLU networks to avoid exponential depth dependence on spectral norms.
• Quantify the trade-offs and identify bottlenecks (notably pre-activation magnitudes) in the bound.
• Provide insights into how training-time properties extend to test data within a PAC-Bayesian setting.

Proposed method

• Introduce input-dependent properties of weights that capture noise-resilience at a given input.
• Define a sequence of conditions (ρ_r,l) and margins ∆⋆_{r,l} that must hold on training data.
• Impose an if-then constraint (Equation 2) ensuring perturbations under Gaussian weight noise remain controlled for inputs satisfying earlier conditions.
• Show how to convert a PAC-Bayes bound on a stochastic network into a bound on the deterministic network (Theorem C.1).
• Specialize to deep ReLU networks; derive a margin-based generalization bound that does not scale with the product of spectral norms (Theorem 4.1).
• Identify pre-activation magnitudes as a bottleneck term (Bpreact) that inversely affects the bound, with discussion on practical mitigations (e.g., ignoring small fractions of data or units).

Experimental results

Research questions

• RQ1 Can training-time noise-resilience properties generalize from training to test data for deep networks?
• RQ2 Can a PAC-Bayesian bound be derived that applies to the original deterministic network rather than a stochastic/compressed version?
• RQ3 Does the resulting bound avoid the conventional exponential depth dependence on the product of spectral norms?
• RQ4 What are the main factors (e.g., pre-activation magnitudes) that govern the tightness of the bound in practice?
• RQ5 How do the theoretical bounds behave empirically with network depth and width on standard datasets like MNIST?

Key findings

• A general PAC-Bayesian framework can bound the test loss of a deterministic, uncompressed network by leveraging training-time noise-resilience.
• For ReLU networks, the bound does not scale with the product of spectral norms and instead depends on interactions between weight matrices and training-time properties.
• The bound scales with depth but at a milder rate (approximately 1.57^D) compared to prior bounds (approximately 2.15^D).
• The main bottleneck is the reciprocal of the smallest training pre-activation magnitude (Bpreact); this can be large if many pre-activations are small, but mitigations (e.g., ignoring outliers) can substantially reduce it.
• Empirical discussion indicates most terms are small (on the order of 10^2) while Bpreact can dominate, highlighting a concrete area for improvement in practice.
• The framework provides a path to non-vacuous guarantees for large networks by focusing on input-dependent properties rather than worst-case spectral-norm products.