[Paper Review] Generalization bounds for deep convolutional neural networks

Philip M. Long, Hanie Sedghi|arXiv (Cornell University)|May 29, 2019

Advanced Neural Network ApplicationsComputer Science43 references33 citations

TL;DR

The paper derives generalization bounds for deep CNNs in terms of distance from initialization and parameter count, independent of input size, with CIFAR-10 experiments validating the bounds.

ABSTRACT

We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss and the distance from the weights to the initial weights. They are independent of the number of pixels in the input, and the height and width of hidden feature maps. We present experiments using CIFAR-10 with varying hyperparameters of a deep convolutional network, comparing our bounds with practical generalization gaps.

Motivation & Objective

• Motivate and analyze generalization for convolutional networks beyond fully connected models.
• Develop bounds that depend on the distance from initialization and total parameter count rather than input dimensions.
• Show that convolutional weight-tying reduces the effective complexity relative to fully connected nets.
• Provide practical implications for initialization and training dynamics in deep CNNs.

Proposed method

• Prove generalization bounds using Lipschitz properties of the network and covering number techniques.
• Define distance measures from initialization for convolutional kernels and last-layer weights.
• Show bounds with a linear depth dependency versus exponential depth in prior work.
• Apply an empirical-process bound (Giné and Guillou) to the covering number-based bound.
• Derive two bound forms: a training/test gap bound and a relative error-type bound.
• Extend bounds to a general CNN+FC architecture with precise Lipschitz parameters.

Experimental results

Research questions

• RQ1 How do generalization bounds for CNNs depend on distance from initialization and parameter count?
• RQ2 Can bounds for CNNs be made independent of input image size and feature-map dimensions?
• RQ3 What is the effect of network depth on the derived generalization bounds?
• RQ4 How do convolutional weight-tying and initialization schemes impact the bounds?
• RQ5 Do empirical experiments (e.g., CIFAR-10) align with the theoretical bounds across hyperparameters?

Key findings

• Bounds scale with the distance from initialization and the total number of parameters, and are independent of input size.
• A linear-in-depth bound is achieved, avoiding exponential depth dependence common in earlier analyses.
• Experiments on CIFAR-10 illustrate that generalization gaps increase with the distance from initialization scaled by parameter count.
• Increasing over-parameterization can flatten generalization gaps in practice, consistent with observed trends, while the distance metric helps explain this behavior.
• The analysis applies to networks combining convolutional and fully connected layers, and aligns with prior fully connected network results.
• The bounds leverage Lipschitz properties of loss and non-expansive activations, alongside covering-number arguments to bound estimation error.