[Paper Review] Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks

Behnam Neyshabur, Zhiyuan Li+3 more|arXiv (Cornell University)|May 30, 2018

Adversarial Robustness in Machine LearningComputer Science19 references214 citations

TL;DR

The paper introduces a unit-wise capacity measure for two-layer ReLU networks that yields a generalization bound decreasing with the number of hidden units, offering an explanation for improved generalization with over-parameterization.

ABSTRACT

Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes, and could potentially explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

Motivation & Objective

• Motivate and quantify why increasing neural network width improves generalization despite over-parameterization.
• Propose a capacity measure based on per-unit capacity and per-unit impact that better matches observed practice.
• Derive a tighter generalization bound for two-layer ReLU networks that decreases with hidden-unit count.
• Provide a matching lower bound on the Rademacher complexity to show tightness of the bound.

Proposed method

• Define unit capacity beta_i as ||u_i - u_i^0||_2 and unit impact alpha_i = ||v_i||_2.
• Restrict the hypothesis class to networks with bounded per-unit capacity and impact (V, U in a set W).
• Compute the Rademacher complexity of the loss composed with this class to obtain a bound that scales with sum_i alpha_i beta_i and data norms.
• Prove a generalization bound (Theorem 2) showing L0(f) ≤ L_gamma_hat(f) + terms shrinking with h, plus a small sqrt(h/m) additive term.
• Provide a lower bound (Theorem 3) for a related class showing the upper bound is tight up to constants.
• Extend discussion to a refined bound for large h using p-norm based covers (Theorem 5 and Corollary 6).

Experimental results

Research questions

• RQ1 Does over-parameterization (increasing hidden units) improve generalization for two-layer ReLU networks, and why?
• RQ2 Can unit-wise capacity and unit-wise impact explain the generalization behavior better than traditional norm-based measures?
• RQ3 What are tight bounds on the Rademacher complexity for two-layer ReLU networks with respect to initialization and per-unit constraints?
• RQ4 How does the proposed bound behave empirically on CIFAR-10, SVHN, and MNIST compared to existing bounds?
• RQ5 Can we extend the results to larger h via p-norm based analysis and what is the trade-off?

Key findings

• The proposed unit-wise capacity bound decreases with increasing hidden units, correlating with lower test error in experiments.
• Per-unit distance to initialization (unit capacity) shrinks with larger networks, while per-unit impact (outgoing weights) also decreases faster than 1/√h.
• The bound uses Frobenius norms of the top layer and the difference between hidden-layer weights and initialization, and shows improvement over existing norm-based bounds.
• A matching lower bound for the Rademacher complexity is established, improving over previous bounds and highlighting a gap beyond Lipschitz-based analyses.
• Empirical results on CIFAR-10 and SVHN show larger networks can generalize better even without explicit regularization, with unit-wise measures decreasing with h.
• A refined bound for extremely large h reduces the problematic sqrt(h) additive term via p-norm based covers (p ≈ log h).