[Paper Review] A statistical model for tensor PCA

Andrea Montanari, Émile Richard|arXiv (Cornell University)|Nov 4, 2014

Sparse and Compressive Sensing Techniques29 references76 citations

TL;DR

This paper develops a statistical model for tensor PCA under a rank-one plus noise assumption, analyzing the trade-off between computational efficiency and statistical accuracy. It establishes that unbounded computation enables recovery when signal-to-noise ratio $\beta \gtrsim \sqrt{k\log k}$, but polynomial-time methods like tensor unfolding and power iteration require $\beta \gtrsim n^{(k-1)/2}$, indicating a computational phase transition.

ABSTRACT

We consider the Principal Component Analysis problem for large tensors of arbitrary order $k$ under a single-spike (or rank-one plus noise) model. On the one hand, we use information theory, and recent results in probability theory, to establish necessary and sufficient conditions under which the principal component can be estimated using unbounded computational resources. It turns out that this is possible as soon as the signal-to-noise ratio $β$ becomes larger than $C\sqrt{k\log k}$ (and in particular $β$ can remain bounded as the problem dimensions increase). On the other hand, we analyze several polynomial-time estimation algorithms, based on tensor unfolding, power iteration and message passing ideas from graphical models. We show that, unless the signal-to-noise ratio diverges in the system dimensions, none of these approaches succeeds. This is possibly related to a fundamental limitation of computationally tractable estimators for this problem. We discuss various initializations for tensor power iteration, and show that a tractable initialization based on the spectrum of the matricized tensor outperforms significantly baseline methods, statistically and computationally. Finally, we consider the case in which additional side information is available about the unknown signal. We characterize the amount of side information that allows the iterative algorithms to converge to a good estimate.

Motivation & Objective

• To understand the fundamental trade-off between computational efficiency and statistical accuracy in tensor PCA.
• To determine the minimal signal-to-noise ratio $\beta$ required for consistent recovery of the underlying rank-one tensor structure.
• To evaluate the performance of polynomial-time algorithms such as tensor unfolding and power iteration in high-dimensional settings.
• To investigate how side information can close the gap between computationally tractable and statistically optimal estimators.
• To characterize the conditions under which message passing and power iteration converge to accurate estimates.

Proposed method

• Uses a spiked tensor model: $\mathbf{X} = \beta \mathbf{v}_0^{\otimes k} + \mathbf{Z}$, where $\mathbf{Z}$ is Gaussian noise.
• Applies information-theoretic arguments to derive a lower bound on $\beta$ for any estimator to succeed, showing $\beta \gtrsim \sqrt{k}$ is necessary.
• Analyzes tensor unfolding by matricizing the tensor into a matrix and applying standard matrix PCA.
• Proposes and analyzes tensor power iteration with a spectral initialization based on the matricized tensor's leading eigenvector.
• Introduces an approximate message passing algorithm and derives its state evolution via a recursive function $f(z;\beta) = \beta^2 (z/(1+z))^{k-1}$.
• Uses a reparametrization $x = \tau^2 / (1 + \tau^2)$ to analyze fixed points of the state evolution and prove convergence to non-trivial solutions when $\beta > \omega_k$.

Experimental results

Research questions

• RQ1 What is the minimal signal-to-noise ratio $\beta$ required for any estimator to consistently recover the spike $\mathbf{v}_0$ in tensor PCA?
• RQ2 Can polynomial-time algorithms such as tensor unfolding or power iteration succeed when $\beta$ remains bounded as $n \to \infty$?
• RQ3 How does side information affect the convergence and accuracy of iterative tensor PCA algorithms?
• RQ4 What is the statistical performance of approximate message passing in tensor PCA, and how does it compare to power iteration?
• RQ5 Is there a computational phase transition in tensor PCA, where tractable algorithms fail even when statistical recovery is possible?

Key findings

• Maximum-likelihood estimation succeeds with high probability when $\beta \geq \mu_k = \sqrt{k\log k}(1 + o_k(1))$, achieving $\|\widehat{\mathbf{v}}^{\text{ML}} - \mathbf{v}_0\|_2^2 \leq 2.01\mu_k / \beta$.
• No estimator can recover $\mathbf{v}_0$ accurately when $\beta \leq c\sqrt{k}$ for a universal constant $c$, establishing a fundamental information-theoretic limit.
• Tensor unfolding succeeds when $\beta \gtrsim n^{(\lceil k/2 \rceil - 1)/2}$, with a conjectured threshold of $\beta \gtrsim n^{(k-2)/4}$ for $k$ even under asymmetric noise.
• Tensor power iteration with spectral initialization converges rapidly to a good estimate when $\beta \gtrsim n^{(k-1)/2}$, and outperforms random initialization.
• Approximate message passing converges to a non-trivial fixed point when $\beta > \omega_k$, and the expected loss is bounded by $6/\beta^2$ in the limit.
• Side information can enable convergence of iterative algorithms even when $\beta$ is below the threshold for standard methods, closing the gap between tractable and optimal estimation.