[Paper Review] Unified Scalable Equivalent Formulations for Schatten Quasi-Norms

Fanhua Shang, Yuanyuan Liu+1 more|arXiv (Cornell University)|Jun 2, 2016

Sparse and Compressive Sensing Techniques31 references10 citations

TL;DR

This paper presents unified, scalable equivalent formulations for Schatten quasi-norms by proving that the Schatten-$p$ quasi-norm of a matrix is equivalent to minimizing the product or weighted sum of Schatten norms of its factor matrices under specific parameter conditions. For $p > 1/2$, the Schatten-$p$ quasi-norm is equivalent to minimizing the Schatten-$2p$ norms of two factor matrices, enabling efficient, smooth optimization without iterative SVD or EVD, thus overcoming the scalability limitations of existing Schatten quasi-norm minimization methods.

ABSTRACT

The Schatten quasi-norm can be used to bridge the gap between the nuclear norm and rank function, and is the tighter approximation to matrix rank. However, most existing Schatten quasi-norm minimization (SQNM) algorithms, as well as for nuclear norm minimization, are too slow or even impractical for large-scale problems, due to the SVD or EVD of the whole matrix in each iteration. In this paper, we rigorously prove that for any p, p1, p2>0 satisfying 1/p=1/p1+1/p2, the Schatten-p quasi-norm of any matrix is equivalent to minimizing the product of the Schatten-p1 norm (or quasi-norm) and Schatten-p2 norm (or quasi-norm) of its two factor matrices. Then we present and prove the equivalence relationship between the product formula of the Schatten quasi-norm and its weighted sum formula for the two cases of p1 and p2: p1=p2 and p1 eq p2. In particular, when p>1/2, there is an equivalence between the Schatten-p quasi-norm of any matrix and the Schatten-2p norms of its two factor matrices, where the widely used equivalent formulation of the nuclear norm can be viewed as a special case. That is, various SQNM problems with p>1/2 can be transformed into the one only involving smooth, convex norms of two factor matrices, which can lead to simpler and more efficient algorithms than conventional methods. We further extend the theoretical results of two factor matrices to the cases of three and more factor matrices, from which we can see that for any 0

Motivation & Objective

• To address the high computational cost of existing Schatten quasi-norm minimization (SQNM) algorithms, which rely on expensive singular value decomposition (SVD) or eigenvalue decomposition (EVD) in each iteration.
• To develop a unified, scalable equivalent formulation for Schatten quasi-norms that avoids non-smooth, non-convex optimization and enables efficient large-scale computation.
• To generalize the known bilinear spectral penalty for nuclear norm (a special case of Schatten-1 norm) to arbitrary Schatten-$p$ quasi-norms with $0 < p < 1$.
• To extend the equivalence from two-factor to three- and multi-factor matrix decompositions, enabling smooth, convex optimization for any $0 < p < 1$.

Proposed method

• Prove that for any $p, p_1, p_2 > 0$ satisfying $1/p = 1/p_1 + 1/p_2$, the Schatten-$p$ quasi-norm of a matrix $X$ is equivalent to minimizing the product of the Schatten-$p_1$ and Schatten-$p_2$ norms of its two factor matrices $U$ and $V$ such that $X = UV^T$.
• Establish equivalence between the product formulation and a weighted sum formulation of the Schatten quasi-norm for both cases $p_1 = p_2$ and $p_1 \neq p_2$.
• Show that when $p > 1/2$, the Schatten-$p$ quasi-norm is equivalent to minimizing the Schatten-$2p$ norms of the two factor matrices, transforming non-smooth, non-convex problems into smooth, convex ones.
• Extend the equivalence to three and more factor matrices, proving that for any $0 < p < 1$, the Schatten-$p$ quasi-norm is equivalent to minimizing the mean of the Schatten-$(\lfloor 1/p \rfloor + 1)p$ norms of all factor matrices.
• Derive closed-form expressions for the equivalence using Hölder's inequality and the method of Lagrange multipliers, ensuring mathematical rigor and generality.
• Demonstrate that the nuclear norm, bi-nuclear, Frobenius/nuclear, and tri-nuclear quasi-norms are special cases of the proposed framework.

Experimental results

Research questions

• RQ1 Can a unified, scalable equivalent formulation be derived for the Schatten-$p$ quasi-norm for any $p > 0$?
• RQ2 Is there a general equivalence between the Schatten-$p$ quasi-norm and the product or weighted sum of Schatten norms of factor matrices under a harmonic mean condition on the exponents?
• RQ3 Can the equivalence be extended from two to three or more factor matrices, and does it preserve smoothness and convexity for $0 < p < 1$?
• RQ4 Does the proposed formulation eliminate the need for iterative SVD or EVD in SQNM, thereby enabling efficient large-scale optimization?
• RQ5 Are well-known quasi-norms such as the nuclear norm, bi-nuclear, and tri-nuclear quasi-norms special cases of the proposed general framework?

Key findings

• For any $p > 1/2$, the Schatten-$p$ quasi-norm of a matrix is equivalent to minimizing the Schatten-$2p$ norms of its two factor matrices, transforming the non-smooth, non-convex SQNM problem into a smooth, convex optimization problem.
• The classical nuclear norm minimization formulation $\|X\|_* = \min_{X=UV^T} (\|U\|_F^2 + \|V\|_F^2)/2$ is a special case of the proposed framework when $p=1$ and $p_1=p_2=2$.
• For any $0 < p < 1$, the Schatten-$p$ quasi-norm is equivalent to minimizing the mean of the Schatten-$(\lfloor 1/p \rfloor + 1)p$ norms of $\lfloor 1/p \rfloor + 1$ factor matrices, enabling scalable optimization via smooth, convex norms.
• The equivalence is rigorously proven using Hölder's inequality and Lagrange multipliers, establishing a general mathematical framework for Schatten quasi-norms.
• The bi-nuclear and Frobenius/nuclear quasi-norms defined in prior work are special cases of the proposed formulation, validating its generality and unifying power.
• The proposed formulations eliminate the need for iterative SVD or EVD in each optimization step, enabling significantly faster and more scalable algorithms for large-scale low-rank matrix recovery problems.