[Paper Review] Implicit Regularization in Deep Learning May Not Be Explainable by Norms

Noam Razin, Nadav Cohen|arXiv (Cornell University)|May 13, 2020

Sparse and Compressive Sensing Techniques86 references46 citations

TL;DR

The paper proves there exist natural matrix factorization problems where gradient flow drives all norms to infinity while the rank minimizes, undermining norm-based explanations of implicit regularization; it emphasizes rank minimization as a more plausible account and provides empirical support including tensor factorization.

ABSTRACT

Mathematically characterizing the implicit regularization induced by gradient-based optimization is a longstanding pursuit in the theory of deep learning. A widespread hope is that a characterization based on minimization of norms may apply, and a standard test-bed for studying this prospect is matrix factorization (matrix completion via linear neural networks). It is an open question whether norms can explain the implicit regularization in matrix factorization. The current paper resolves this open question in the negative, by proving that there exist natural matrix factorization problems on which the implicit regularization drives all norms (and quasi-norms) towards infinity. Our results suggest that, rather than perceiving the implicit regularization via norms, a potentially more useful interpretation is minimization of rank. We demonstrate empirically that this interpretation extends to a certain class of non-linear neural networks, and hypothesize that it may be key to explaining generalization in deep learning.

Motivation & Objective

• Motivate and analyze the mystery of implicit regularization in gradient-based deep learning.
• Disprove the idea that implicit regularization is captured by minimizing norms for matrix factorization.
• Show that there are problems where all norms grow unbounded while rank decreases.
• Extend the investigation to tensor factorization to assess generality and potential applicability to deep models.

Proposed method

• Model deep matrix factorization with depth L ≥ 2 and unconstrained hidden dimensions.
• Analyze gradient flow dynamics of the overparameterized objective that fits observed matrix entries.
• Use the product W_{L:1} = W_L W_{L-1} ... W_1 to represent the input–output mapping (Equation 3).
• Assume balanced initialization to derive differential equations governing W_{L:1} and its singular values.
• Establish theoretical bounds showing norms grow as loss decreases while effective rank shrinks (Theorem 1).
• Empirically corroborate findings and extend the investigation to tensor factorization.

Experimental results

Research questions

• RQ1 Can gradient-based optimization of deep matrix factorization be characterized by minimization of matrix norms (or quasi-norms) as implicit regularization?
• RQ2 Are there natural matrix completion settings where all norms grow to infinity while rank is minimized, contradicting norm-based explanations?
• RQ3 Does a rank-focused interpretation (rather than norm minimization) better explain generalization in linear/non-linear architectures?
• RQ4 Do similar implicit regularization phenomena extend to tensor factorization and related non-linear networks?
• RQ5 What is the robustness of these phenomena to perturbations in observed entries and initialization?

Key findings

• There exist matrix completion problems where gradient flow on deep matrix factorizations drives all norms toward infinity while reducing rank.
• This behavior supports Conjecture 2, showing norms are not minimized and can diverge, in contrast to norm-based explanations.
• The product matrix's positive determinant at initialization leads to norm growth and rank minimization during loss reduction (Theorem 1).
• The phenomenon holds with probability 0.5 under common random initializations (Proposition 3).
• Corollary 1 states that global loss minimization implies norms diverge to infinity while effective rank and distance to infimal rank indicate rank minimization.
• Robustness results (Theorem 2) show the behavior persists under perturbations of observed values and locations.