[Paper Review] Limitations of Implicit Bias in Matrix Sensing: Initialization Rank Matters

Armin Eftekhari, Konstantinos C. Zygalakis|arXiv (Cornell University)|Aug 27, 2020

Sparse and Compressive Sensing Techniques38 references4 citations

TL;DR

This paper identifies initialization rank as a critical limitation of implicit bias in matrix sensing via gradient flow, showing that low-rank initialization enhances recovery of a planted low-rank matrix even when the initialization norm is large. It establishes a theoretical capture neighborhood for successful recovery and proposes an alternative algorithm that complements existing high-rank, near-zero initialization schemes.

ABSTRACT

In matrix sensing, we first numerically identify the sensitivity to the initialization rank as a new limitation of the implicit bias of gradient flow. We will partially quantify this phenomenon mathematically, where we establish that the gradient flow of the empirical risk is implicitly biased towards low-rank outcomes and successfully learns the planted low-rank matrix, provided that the initialization is low-rank and within a specific "capture neighborhood". This capture neighborhood is far larger than the corresponding neighborhood in local refinement results; the former contains all models with zero training error whereas the latter is a small neighborhood of a model with zero test error. These new insights enable us to design an alternative algorithm for matrix sensing that complements the high-rank and near-zero initialization scheme which is predominant in the existing literature.

Motivation & Objective

• To investigate how initialization rank influences the implicit bias of gradient flow in matrix sensing.
• To identify limitations of existing implicit bias theories that assume near-zero or high-rank initialization.
• To theoretically quantify the capture neighborhood where gradient flow successfully recovers the planted low-rank matrix.
• To propose a new algorithmic framework that leverages low-rank initialization to improve recovery performance.
• To demonstrate empirically that implicit bias diminishes with increasing initialization norm and rank, especially when far from origin.

Proposed method

• Formalize the gradient flow dynamics on the factorized matrix space: $ \dot{U}(t) = -\nabla \| \mathcal{A}(U(t)U(t)^T) - b \|_2^2 $.
• Analyze the trajectory of gradient flow under low-rank vs. high-rank initialization, particularly focusing on proximity to the planted low-rank matrix $ X^\natural $.
• Define a 'capture neighborhood' where gradient flow converges to $ X^\natural $, showing it is larger than local refinement neighborhoods.
• Use singular value decomposition and Frobenius norm identities to derive lower bounds on the distance between factorized matrices.
• Establish a key inequality: $ \| UU^T - VV^T \|_F^2 \geq (\max(\sigma_{\min}(U), \sigma_{\min}(V)))^2 \cdot \text{dist}(U, V\mathcal{O}_p)^2 $, linking matrix distance to singular values and alignment.
• Leverage the nuclear norm and orthogonal invariance to relate the alignment of singular vectors to the recovery error.

Experimental results

Research questions

• RQ1 How does initialization rank affect the implicit bias of gradient flow in matrix sensing?
• RQ2 Can the capture neighborhood for successful recovery be quantified, and how does it compare to local refinement neighborhoods?
• RQ3 Why does gradient flow fail to recover the planted low-rank matrix when initialized far from the origin or with high rank?
• RQ4 What theoretical guarantees can be provided for gradient flow when initialized with low-rank structures?
• RQ5 Can a new algorithm be designed that improves recovery by exploiting low-rank initialization, independent of initialization norm?

Key findings

• Initialization rank significantly influences implicit bias: low-rank initialization leads to stronger bias toward the planted low-rank matrix than high-rank initialization.
• The capture neighborhood for successful recovery under gradient flow is larger than local refinement neighborhoods, encompassing all models with zero training error.
• Gradient flow fails to recover the planted matrix when initialized far from the origin, even with low rank, indicating norm sensitivity.
• Theoretical analysis shows that $ \| UU^T - VV^T \|_F^2 \geq (\max(\sigma_{\min}(U), \sigma_{\min}(V)))^2 \cdot \text{dist}(U, V\mathcal{O}_p)^2 $, linking matrix distance to singular values and alignment.
• Numerical experiments confirm that test error increases with both initialization norm and rank, with low-rank initialization consistently outperforming high-rank counterparts.