[Paper Review] Spectrally-normalized margin bounds for neural networks

Peter L. Bartlett, Dylan J. Foster+1 more|arXiv (Cornell University)|Jun 26, 2017

Neural Networks and Applications2 references173 citations

TL;DR

The paper derives a margin-based generalization bound for multiclass neural networks that scales with the margin-normalized spectral complexity (product of spectral norms times a correction term) and provides empirical support using AlexNet on MNIST and CIFAR-10. The bound remains multiclass and does not explicitly depend on the number of layers or units, apart from log factors, and is validated via margin distributions during training.

ABSTRACT

This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized "spectral complexity": their Lipschitz constant, meaning the product of the spectral norms of the weight matrices, times a certain correction factor. This bound is empirically investigated for a standard AlexNet network trained with SGD on the mnist and cifar10 datasets, with both original and random labels; the bound, the Lipschitz constants, and the excess risks are all in direct correlation, suggesting both that SGD selects predictors whose complexity scales with the difficulty of the learning task, and secondly that the presented bound is sensitive to this complexity.

Motivation & Objective

• Develop a margin-based generalization bound for multiclass neural networks that scales with margin-normalized spectral complexity.
• Show that the bound depends on spectral norms and a correction factor rather than combinatorial network parameters.
• Empirically validate the bound using AlexNet-like architectures on MNIST and CIFAR datasets, including with random labels.
• Demonstrate how margin normalization aligns with generalization dynamics and task difficulty.

Proposed method

• Define spectral complexity R_A as the product of layer-wise spectral norms times a margin-normalized correction term (involving A_i^T - M_i^T and a 2,1 norm).
• Prove a multiclass margin bound (Theorem 1.1) showing Pr[argmax F_A(x) ≠ y] ≤ R̂_γ(F_A) + Õ((||X||_2 R_A)/(γ n) log(W) + sqrt(log(1/δ)/n)).
• Use covering-number arguments and Maurey sparsification to bound the Rademacher complexity of the network class, leading to the main bound.
• Relate the bound to a margin distribution and show it remains informative under growing weight norms.
• Provide a lower bound for the Rademacher complexity to illustrate tightness in part of the analysis.

Experimental results

Research questions

• RQ1 How can one formulate a margin-based generalization bound for multiclass neural networks that scales with margin-normalized spectral complexity?
• RQ2 Does a bound that depends on spectral norms and a margin normalization provide meaningful guidance for generalization across different tasks and label configurations?
• RQ3 Can the bound be made independent of explicit combinatorial network parameters like depth or width beyond logarithmic factors?
• RQ4 Do margin distributions observed during training correlate with excess risk and reflect task difficulty?
• RQ5 What empirical evidence supports the bound's relevance on standard datasets and with randomized labels?

Key findings

• A margin-based bound is established that scales with the product of spectral norms divided by the margin, plus a correction term, and is multiclass without explicit dependence on the number of classes.
• The bound relies on spectral complexity R_A and reference matrices M_i, capturing distance from the reference network.
• Empirical analysis on AlexNet-like networks shows margin distributions correlate with task difficulty and with excess risk across MNIST, CIFAR-10, and label/random-label settings.
• Margin distributions converge during training even as weight norms grow, and l2 regularization does not strongly affect margins or generalization in these experiments.
• Regularization that meaningfully improves margins is identified as an open problem, suggesting a gap between typical weight decay and margin optimization.