[Paper Review] In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Behnam Neyshabur, Ryota Tomioka+1 more|arXiv (Cornell University)|Dec 20, 2014

Neural Networks and ApplicationsComputer Science134 citations

TL;DR

This paper argues that implicit regularization—specifically, implicit $β$-norm regularization in stochastic gradient descent—acts as the true inductive bias in deep learning, not network size. Through an analogy to matrix factorization, it shows that weight decay in large, overparameterized networks leads to an equivalent convex neural network with $β$-regularization, explaining generalization despite large capacity.

ABSTRACT

We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.

Motivation & Objective

• To challenge the assumption that network size is the primary capacity control in deep learning.
• To identify the true inductive bias enabling generalization in overparameterized deep networks.
• To establish a theoretical link between implicit regularization in SGD and convex neural networks.
• To demonstrate that implicit $β$-regularization in large networks is equivalent to $β$-regularization in an infinite-width convex neural net.
• To show that weight decay in overparameterized networks leads to a solution equivalent to group lasso regularization in a convex formulation.

Proposed method

• Empirically evaluates training and test error across increasing network sizes to observe generalization behavior.
• Draws analogy between deep learning and matrix factorization to identify implicit regularization as the key capacity control.
• Uses the equivalence between weight decay and $β$-regularization in overparameterized networks to derive a convex neural net formulation.
• Derives that minimizing weight decay in a large network is equivalent to solving a convex optimization problem with $β$-regularization on the output layer weights.
• Establishes that the infinite-width limit of a two-layer ReLU network with weight decay yields a convex neural net with $β$-regularization.
• Proves that the solution of a large, weight-decayed network is equivalent to a convex neural net with group lasso regularization when $H > n$.

Experimental results

Research questions

• RQ1 What mechanism enables generalization in overparameterized deep neural networks if not network size?
• RQ2 How does implicit regularization in SGD relate to explicit regularization in convex models?
• RQ3 Can the inductive bias in deep learning be characterized as a form of norm regularization rather than architectural capacity?
• RQ4 Is there a convex optimization formulation equivalent to training a large, weight-decayed deep network?
• RQ5 What is the role of the input-to-hidden weight norm in determining the inductive bias of deep networks?

Key findings

• Increasing network size does not improve generalization beyond a certain point, indicating that size is not the primary capacity control.
• Implicit $β$-regularization via SGD in large networks leads to solutions equivalent to $β$-regularized convex neural networks.
• The equivalence holds when the number of hidden units $H$ exceeds the number of training samples $n$, ensuring the solution is sparse in the hidden unit space.
• Weight decay in a large network is equivalent to $β$-regularization on the output layer weights in a convex neural net with an infinite library of hidden units.
• The convex neural net formulation with $β$-regularization achieves the same generalization performance as training a large, weight-decayed network.
• The solution of the convex neural net is always discrete with at most $n+1$ active hidden units, confirming the sparsity induced by implicit regularization.