[论文解读] Spectrally-normalized margin bounds for neural networks

Peter L. Bartlett, Dylan J. Foster+1 more|arXiv (Cornell University)|Jun 26, 2017

Neural Networks and Applications参考文献 2被引用 173

一句话总结

本论文推导了一种基于边际的多类神经网络泛化界，随着边际归一化的谱复杂度（谱范数的乘积乘以一个校正项）而扩展，并在 MNIST 和 CIFAR-10 上用 AlexNet 进行经验支持。界限保持多类性质，除了对数因子外，不直接依赖层数或单元数，并通过训练过程中的边际分布进行验证。

ABSTRACT

This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized "spectral complexity": their Lipschitz constant, meaning the product of the spectral norms of the weight matrices, times a certain correction factor. This bound is empirically investigated for a standard AlexNet network trained with SGD on the mnist and cifar10 datasets, with both original and random labels; the bound, the Lipschitz constants, and the excess risks are all in direct correlation, suggesting both that SGD selects predictors whose complexity scales with the difficulty of the learning task, and secondly that the presented bound is sensitive to this complexity.

研究动机与目标

• Develop a margin-based generalization bound for multiclass neural networks that scales with margin-normalized spectral complexity.
• Show that the bound depends on spectral norms and a correction factor rather than combinatorial network parameters.
• Empirically validate the bound using AlexNet-like architectures on MNIST and CIFAR datasets, including with random labels.
• Demonstrate how margin normalization aligns with generalization dynamics and task difficulty.

提出的方法

• Define spectral complexity R_A as the product of layer-wise spectral norms times a margin-normalized correction term (involving A_i^T - M_i^T and a 2,1 norm).
• Prove a multiclass margin bound (Theorem 1.1) showing Pr[argmax F_A(x) ≠ y] ≤ R̂_γ(F_A) + Õ((||X||_2 R_A)/(γ n) log(W) + sqrt(log(1/δ)/n)).
• Use covering-number arguments and Maurey sparsification to bound the Rademacher complexity of the network class, leading to the main bound.
• Relate the bound to a margin distribution and show it remains informative under growing weight norms.
• Provide a lower bound for the Rademacher complexity to illustrate tightness in part of the analysis.

实验结果

研究问题

• RQ1 How can one formulate a margin-based generalization bound for multiclass neural networks that scales with margin-normalized spectral complexity?
• RQ2 Does a bound that depends on spectral norms and a margin normalization provide meaningful guidance for generalization across different tasks and label configurations?
• RQ3 Can the bound be made independent of explicit combinatorial network parameters like depth or width beyond logarithmic factors?
• RQ4 Do margin distributions observed during training correlate with excess risk and reflect task difficulty?
• RQ5 What empirical evidence supports the bound's relevance on standard datasets and with randomized labels?

主要发现

• A margin-based bound is established that scales with the product of spectral norms divided by the margin, plus a correction term, and is multiclass without explicit dependence on the number of classes.
• The bound relies on spectral complexity R_A and reference matrices M_i, capturing distance from the reference network.
• Empirical analysis on AlexNet-like networks shows margin distributions correlate with task difficulty and with excess risk across MNIST, CIFAR-10, and label/random-label settings.
• Margin distributions converge during training even as weight norms grow, and l2 regularization does not strongly affect margins or generalization in these experiments.
• Regularization that meaningfully improves margins is identified as an open problem, suggesting a gap between typical weight decay and margin optimization.