[Paper Review] Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

Noam Razin, Asaf Maman+1 more|arXiv (Cornell University)|Jan 27, 2022

Tensor decomposition and applications4 citations

TL;DR

This paper establishes that gradient descent in hierarchical tensor factorization — a model equivalent to deep convolutional neural networks — induces implicit regularization toward low hierarchical tensor rank, which corresponds to locality in the associated networks. Theoretical analysis reveals a momentum-like dynamics in local components, and empirical validation shows that explicitly discouraging locality improves performance on non-local tasks, challenging conventional architectural design principles.

ABSTRACT

In the pursuit of explaining implicit regularization in deep learning, prominent focus was given to matrix and tensor factorizations, which correspond to simplified neural networks. It was shown that these models exhibit an implicit tendency towards low matrix and tensor ranks, respectively. Drawing closer to practical deep learning, the current paper theoretically analyzes the implicit regularization in hierarchical tensor factorization, a model equivalent to certain deep convolutional neural networks. Through a dynamical systems lens, we overcome challenges associated with hierarchy, and establish implicit regularization towards low hierarchical tensor rank. This translates to an implicit regularization towards locality for the associated convolutional networks. Inspired by our theory, we design explicit regularization discouraging locality, and demonstrate its ability to improve the performance of modern convolutional networks on non-local tasks, in defiance of conventional wisdom by which architectural changes are needed. Our work highlights the potential of enhancing neural networks via theoretical analysis of their implicit regularization.

Motivation & Objective

• To theoretically analyze implicit regularization in hierarchical tensor factorization, a model equivalent to deep convolutional neural networks.
• To understand why convolutional networks struggle with long-range dependencies, beyond just expressive capacity limitations.
• To demonstrate that the implicit bias toward low hierarchical tensor rank enforces locality in learned representations.
• To design and validate explicit regularization that counteracts this implicit bias, improving performance on non-local tasks.

Proposed method

• Analyzes gradient descent dynamics in hierarchical tensor factorization using a dynamical systems approach.
• Identifies local components within the hierarchical structure and characterizes their evolution under small learning rates and near-zero initialization.
• Establishes a momentum-like effect on local components, causing incremental learning that favors low hierarchical tensor rank.
• Links hierarchical tensor rank to locality in convolutional networks, where low rank implies short-range dependencies.
• Proposes explicit regularization to discourage locality by penalizing high-rank components in the hierarchical factorization.
• Validates the approach empirically on modern convolutional networks, showing improved performance on non-local tasks.

Experimental results

Research questions

• RQ1 What implicit inductive bias does gradient descent induce in hierarchical tensor factorization?
• RQ2 How does hierarchical tensor rank relate to locality in the corresponding convolutional neural networks?
• RQ3 Can the implicit bias toward locality be explicitly counteracted to improve performance on non-local tasks?
• RQ4 Does the momentum-like dynamics of local components in hierarchical factorization explain low-rank solutions?
• RQ5 What is the theoretical mechanism behind implicit regularization in deep, non-linear, hierarchical models?

Key findings

• Gradient descent in hierarchical tensor factorization induces implicit regularization toward low hierarchical tensor rank due to a momentum-like dynamics in local components.
• Low hierarchical tensor rank corresponds to a bias toward modeling only short-range (local) dependencies in the input, explaining the poor performance of CNNs on long-range tasks.
• Explicit regularization that discourages locality improves the performance of modern convolutional networks on non-local tasks, defying the conventional need for architectural changes.
• The theoretical analysis confirms that the implicit bias is not due to norm minimization, but rather due to incremental learning driven by component magnitude dynamics.
• Empirical results validate that anti-locality regularization enhances generalization on tasks requiring long-range reasoning, such as image generation and semantic segmentation.