[论文解读] Implicit Regularization in Deep Learning May Not Be Explainable by Norms

Noam Razin, Nadav Cohen|arXiv (Cornell University)|May 13, 2020

Sparse and Compressive Sensing Techniques参考文献 86被引用 46

一句话总结

本论文证明存在自然矩阵分解问题，其中梯度流动将所有范数推向无穷大，而秩在减小，动摇了基于范数的隐式正则化解释；它强调秩最小化作为更可信的解释，并提供包括张量分解在内的实证支持。

ABSTRACT

Mathematically characterizing the implicit regularization induced by gradient-based optimization is a longstanding pursuit in the theory of deep learning. A widespread hope is that a characterization based on minimization of norms may apply, and a standard test-bed for studying this prospect is matrix factorization (matrix completion via linear neural networks). It is an open question whether norms can explain the implicit regularization in matrix factorization. The current paper resolves this open question in the negative, by proving that there exist natural matrix factorization problems on which the implicit regularization drives all norms (and quasi-norms) towards infinity. Our results suggest that, rather than perceiving the implicit regularization via norms, a potentially more useful interpretation is minimization of rank. We demonstrate empirically that this interpretation extends to a certain class of non-linear neural networks, and hypothesize that it may be key to explaining generalization in deep learning.

研究动机与目标

• Motivate and analyze the mystery of implicit regularization in gradient-based deep learning.
• Disprove the idea that implicit regularization is captured by minimizing norms for matrix factorization.
• Show that there are problems where all norms grow unbounded while rank decreases.
• Extend the investigation to tensor factorization to assess generality and potential applicability to deep models.

提出的方法

• Model deep matrix factorization with depth L ≥ 2 and unconstrained hidden dimensions.
• Analyze gradient flow dynamics of the overparameterized objective that fits observed matrix entries.
• Use the product W_{L:1} = W_L W_{L-1} ... W_1 to represent the input–output mapping (Equation 3).
• Assume balanced initialization to derive differential equations governing W_{L:1} and its singular values.
• Establish theoretical bounds showing norms grow as loss decreases while effective rank shrinks (Theorem 1).
• Empirically corroborate findings and extend the investigation to tensor factorization.

实验结果

研究问题

• RQ1 Can gradient-based optimization of deep matrix factorization be characterized by minimization of matrix norms (or quasi-norms) as implicit regularization?
• RQ2 Are there natural matrix completion settings where all norms grow to infinity while rank is minimized, contradicting norm-based explanations?
• RQ3 Does a rank-focused interpretation (rather than norm minimization) better explain generalization in linear/non-linear architectures?
• RQ4 Do similar implicit regularization phenomena extend to tensor factorization and related non-linear networks?
• RQ5 What is the robustness of these phenomena to perturbations in observed entries and initialization?

主要发现

• There exist matrix completion problems where gradient flow on deep matrix factorizations drives all norms toward infinity while reducing rank.
• This behavior supports Conjecture 2, showing norms are not minimized and can diverge, in contrast to norm-based explanations.
• The product matrix's positive determinant at initialization leads to norm growth and rank minimization during loss reduction (Theorem 1).
• The phenomenon holds with probability 0.5 under common random initializations (Proposition 3).
• Corollary 1 states that global loss minimization implies norms diverge to infinity while effective rank and distance to infimal rank indicate rank minimization.
• Robustness results (Theorem 2) show the behavior persists under perturbations of observed values and locations.