[Paper Review] Interpolating Convex and Non-Convex Tensor Decompositions via the Subspace Norm

Qinqing Zheng, Ryota Tomioka|arXiv (Cornell University)|Mar 18, 2015

Sparse and Compressive Sensing Techniques21 references3 citations

TL;DR

This paper introduces the subspace norm, a novel tensor denoising method that interpolates between convex and non-convex tensor decomposition techniques by leveraging Kronecker products of mode-wise singular vectors from truncated SVDs of tensor unfoldings. It achieves a nearly ideal signal-to-noise ratio bound of $O(\sqrt{n} + \sqrt{H^{K-1}})$, significantly improving upon prior methods, and demonstrates empirically near-optimal performance even for small $H$, such as $H=O(1)$.

ABSTRACT

We consider the problem of recovering a low-rank tensor from its noisy observation. Previous work has shown a recovery guarantee with signal to noise ratio $O(n^{\lceil K/2 ceil /2})$ for recovering a $K$th order rank one tensor of size $n imes \cdots imes n$ by recursive unfolding. In this paper, we first improve this bound to $O(n^{K/4})$ by a much simpler approach, but with a more careful analysis. Then we propose a new norm called the subspace norm, which is based on the Kronecker products of factors obtained by the proposed simple estimator. The imposed Kronecker structure allows us to show a nearly ideal $O(\sqrt{n}+\sqrt{H^{K-1}})$ bound, in which the parameter $H$ controls the blend from the non-convex estimator to mode-wise nuclear norm minimization. Furthermore, we empirically demonstrate that the subspace norm achieves the nearly ideal denoising performance even with $H=O(1)$.

Motivation & Objective

• To close the gap between computationally tractable tensor denoising methods and the information-theoretically optimal performance.
• To develop a unified framework that interpolates between non-convex estimators (e.g., alternating minimization) and convex norms (e.g., latent trace norm).
• To improve the signal-to-noise ratio (SNR) threshold required for reliable low-rank tensor recovery beyond existing recursive unfolding and unfolding-based methods.
• To design a more efficient estimator by reducing the SVD computation size from $n \times n^{K-1}$ to $n \times H^{K-1}$.

Proposed method

• Proposes the subspace norm as a convex relaxation that combines Kronecker products of orthonormal matrices $\widehat{\boldsymbol{P}}^{(k)}$ estimated via mode-wise truncated SVD on tensor unfoldings.
• Uses the higher-order singular value decomposition (HOSVD) to extract low-dimensional factor subspaces for each mode of the tensor.
• Constructs the prediction as a mixture of $K$ low-rank tensors, each formed by folding a matrix product involving Kronecker products of estimated factor matrices from other modes.
• Employs a user-defined parameter $H$ to control the trade-off between non-convexity (small $H$) and convexity (large $H$), with $H=n$ recovering the latent trace norm.
• Analyzes the non-asymptotic behavior of the leading singular vector from ordinary unfolding to derive a tighter SNR bound of $O(n^{K/4})$.
• Uses random matrix theory and concentration inequalities to bound the spectral norm of a noise matrix, establishing high-probability recovery guarantees.

Experimental results

Research questions

• RQ1 Can a convex relaxation be designed that achieves nearly optimal denoising performance for low-rank tensors while remaining computationally efficient?
• RQ2 What is the minimal signal-to-noise ratio required for reliable recovery of a $K$th-order rank-one tensor using a simple, non-recursive unfolding-based estimator?
• RQ3 How can the strengths of non-convex optimization (e.g., alternating minimization) and convex relaxation (e.g., nuclear norm) be unified in tensor decomposition?
• RQ4 To what extent does the performance of the proposed norm degrade as the parameter $H$ is reduced from $n$ to a constant?

Key findings

• The proposed method achieves a nearly ideal signal-to-noise ratio bound of $O(\sqrt{n} + \sqrt{H^{K-1}})$, which interpolates between the intractable ideal estimator and convex relaxation methods.
• The analysis shows a two-phase decay in estimation error: fast $1/\beta^4$ decay when $n^{K/4} \lesssim \beta/\sigma \lesssim n^{K/2}$, and slower $1/\beta^2$ decay beyond $n^{K/2}$.
• The subspace norm reduces the required SVD computation size from $n \times n^{K-1}$ to $n \times H^{K-1}$, significantly improving computational efficiency.
• Empirical results confirm that the subspace norm achieves near-optimal denoising performance even for small $H$, such as $H=O(1)$, suggesting strong practical utility.
• The method improves the SNR threshold for recovery from $O(n^{\lceil K/2\rceil/2})$ (recursive unfolding) to $O(n^{K/4})$ via a simpler analysis.
• When $H=n$, the subspace norm reduces to the latent trace norm, confirming consistency with prior convex approaches.