[Paper Review] Generalization in Deep Networks: The Role of Distance from Initialization

Vaishnavh Nagarajan, J. Zico Kolter|arXiv (Cornell University)|Jan 7, 2019

Stochastic Gradient Optimization TechniquesComputer Science12 references57 citations

TL;DR

The paper proposes initialization-dependent effective model capacity and provides empirical and theoretical evidence that SGD implicitly regularizes the distance from initialization, which helps explain generalization in deep networks.

ABSTRACT

Why does training deep neural networks using stochastic gradient descent (SGD) result in a generalization error that does not worsen with the number of parameters in the network? To answer this question, we advocate a notion of effective model capacity that is dependent on {\em a given random initialization of the network} and not just the training algorithm and the data distribution. We provide empirical evidences that demonstrate that the model capacity of SGD-trained deep networks is in fact restricted through implicit regularization of {\em the $\ell_2$ distance from the initialization}. We also provide theoretical arguments that further highlight the need for initialization-dependent notions of model capacity. We leave as open questions how and why distance from initialization is regularized, and whether it is sufficient to explain generalization.

Motivation & Objective

• Motivate an effective model capacity that depends on a fixed random initialization of the network.
• Empirically show that SGD regularizes the distance from initialization to a width-independent value.
• Theoretically argue that initialization-dependent measures are needed to explain generalization beyond width-based norms.
• Explore the limitations of initialization-insensitive norms (e.g., distance from origin) for explaining generalization.

Proposed method

• Define the effective capacity H_{m,elta}[D,(Z,C),A] as the set of parameter configurations SGD can reach with high probability.
• Empirically analyze how the distance r from initialization behaves as network width H and training set size m vary, on real and corrupted labels.
• Show that r stays width-independent for moderate H and grows with training data size m and label noise.
• Provide theoretical results for linear networks showing Rademacher complexity bounds for networks within fixed distance r from initialization.
• Discuss why norms that measure distance to origin may fail to explain generalization.
• Present corollaries and lemmas that connect initialization, distance regularization, and generalization bounds.

Experimental results

Research questions

• RQ1 Does the effective model capacity of a neural network depend on the random initialization?
• RQ2 Can SGD implicitly regularize the distance from initialization to a width-independent value, and does this relate to generalization?
• RQ3 Are initialization-dependent measures more informative than distance-from-origin norms for explaining generalization?
• RQ4 What theoretical guarantees can be established for networks constrained to lie within a fixed distance from initialization?
• RQ5 How does distance from initialization scale with width, data size, and label noise?

Key findings

• Distance moved from initialization is largely width-independent for moderate widths and tends to stay bounded as width grows.
• Distance r increases with label noise and larger training set size, indicating more memorization is needed for noisier data.
• For linear networks, the Rademacher complexity of networks within an initialization-based distance r has a width-independent bound (scales with r and network depth).
• Norms tied to distance from the origin may grow with width and fail to explain generalization, highlighting the value of initialization-dependent measures.
• Random initialization alone yields width-independent generalization error for the untrained network, supporting initialization-based capacity concepts.