[Paper Review] Guaranteed Non-Orthogonal Tensor Decomposition via Alternating Rank-$1$ Updates

Animashree Anandkumar, Rong Ge+1 more|arXiv (Cornell University)|Feb 21, 2014

Tensor decomposition and applications49 references96 citations

TL;DR

This paper proposes a guaranteed non-orthogonal tensor decomposition algorithm using alternating rank-1 updates, achieving local and global convergence for third-order tensors under incoherence and rank conditions. It provides linear convergence for $k = o(d^{1.5})$ and global convergence under $k \leq \beta d$, with tight perturbation bounds for noisy tensors, enabling overcomplete representations without whitening.

ABSTRACT

In this paper, we provide local and global convergence guarantees for recovering CP (Candecomp/Parafac) tensor decomposition. The main step of the proposed algorithm is a simple alternating rank-$1$ update which is the alternating version of the tensor power iteration adapted for asymmetric tensors. Local convergence guarantees are established for third order tensors of rank $k$ in $d$ dimensions, when $k=o \bigl( d^{1.5} \bigr)$ and the tensor components are incoherent. Thus, we can recover overcomplete tensor decomposition. We also strengthen the results to global convergence guarantees under stricter rank condition $k \le βd$ (for arbitrary constant $β> 1$) through a simple initialization procedure where the algorithm is initialized by top singular vectors of random tensor slices. Furthermore, the approximate local convergence guarantees for $p$-th order tensors are also provided under rank condition $k=o \bigl( d^{p/2} \bigr)$. The guarantees also include tight perturbation analysis given noisy tensor.

Motivation & Objective

• To provide convergence guarantees for non-orthogonal CP tensor decomposition without requiring whitening, which is computationally expensive and numerically unstable.
• To enable recovery of overcomplete tensor decompositions where the number of components $k$ exceeds the dimension $d$, by relaxing orthogonality constraints.
• To establish global convergence through a simple initialization using top singular vectors of random tensor slices, avoiding reliance on strong assumptions.
• To provide tight perturbation analysis for noisy tensors, ensuring robustness in practical applications.
• To demonstrate that incoherent tensor components—common in latent variable models—enable efficient and stable decomposition via rank-1 updates.

Proposed method

• The algorithm performs alternating rank-1 updates along each tensor mode, projecting the current estimate onto the current mode using the other modes' current estimates.
• Each update step is derived from a tensor power iteration variant adapted for asymmetric, non-orthogonal tensors.
• The method avoids whitening by relying on incoherence of tensor components, which acts as a soft orthogonality constraint.
• A novel initialization procedure uses top singular vectors from random tensor slices to achieve global convergence under $k \leq \beta d$.
• Theoretical analysis leverages incoherence assumptions and 3-norm concentration to bound approximation error and convergence rate.
• Perturbation analysis is derived using tensor norm inequalities and stability bounds under noise, showing error scales as $O(\epsilon^{3-p})$ for $p$-th order tensors.

Experimental results

Research questions

• RQ1 Can alternating rank-1 updates achieve local convergence for non-orthogonal tensor decomposition under incoherence and sub-cubic rank scaling?
• RQ2 Does a simple initialization via random tensor slice singular vectors enable global convergence for rank-$k$ tensors with $k \leq \beta d$?
• RQ3 Can the algorithm recover overcomplete tensor decompositions where $k = o(d^{1.5})$ without whitening or orthogonality constraints?
• RQ4 How does the algorithm perform under noisy tensor inputs, and what is the tightness of the perturbation bound?
• RQ5 What is the relationship between incoherence, approximation error, and convergence rate in alternating rank-1 updates?

Key findings

• Local linear convergence is achieved for third-order tensors with $k = o(d^{1.5})$ under incoherence, with convergence rate dependent on $\tilde{O}(\sqrt{k}/d)$ error in component recovery.
• Global convergence is guaranteed under the stricter rank condition $k \leq \beta d$ for any constant $\beta > 1$, using a top-singular-vector initialization.
• The algorithm recovers components within $\tilde{O}(w_{\max}\sqrt{k}/(w_{\min}d))$ distance of the true components, with error scaling controlled by incoherence.
• For noisy tensors, the perturbation error scales as $O(\epsilon^{3-p})$ for $p$-th order tensors, with tight bounds derived via 3-norm concentration.
• The method achieves robustness to noise and avoids ill-conditioning, unlike whitening-based approaches, while maintaining computational efficiency.
• Theoretical guarantees extend to $p$-th order tensors under $k = o(d^{p/2})$, demonstrating scalability to higher-order decompositions.