[Paper Review] Tensor Completion Made Practical

Allen Liu, Ankur Moitra|arXiv (Cornell University)|Jun 4, 2020

Tensor decomposition and applications30 references4 citations

TL;DR

This paper proposes a novel variant of alternating minimization for tensor completion that achieves linear convergence to the true tensor even under high factor correlation, with provable guarantees and nearly linear runtime. Unlike standard alternating minimization, which can converge slowly or diverge in practice, the new method—Kronecker Completion—uses a structured update rule inspired by progress measures from matrix completion, enabling fast, stable recovery of third-order tensors from sparse observations, including in high-dimensional settings (e.g., 1000×1000×1000) with minimal data.

ABSTRACT

Tensor completion is a natural higher-order generalization of matrix completion where the goal is to recover a low-rank tensor from sparse observations of its entries. Existing algorithms are either heuristic without provable guarantees, based on solving large semidefinite programs which are impractical to run, or make strong assumptions such as requiring the factors to be nearly orthogonal. In this paper we introduce a new variant of alternating minimization, which in turn is inspired by understanding how the progress measures that guide convergence of alternating minimization in the matrix setting need to be adapted to the tensor setting. We show strong provable guarantees, including showing that our algorithm converges linearly to the true tensors even when the factors are highly correlated and can be implemented in nearly linear time. Moreover our algorithm is also highly practical and we show that we can complete third order tensors with a thousand dimensions from observing a tiny fraction of its entries. In contrast, and somewhat surprisingly, we show that the standard version of alternating minimization, without our new twist, can converge at a drastically slower rate in practice.

Motivation & Objective

• Address the gap between theoretical tensor completion algorithms—often impractical due to large semidefinite programs—and practical methods like alternating minimization, which lack rigorous convergence guarantees.
• Overcome the limitations of standard alternating minimization, which can converge slowly or exhibit pathological behavior (e.g., slower convergence with more observations) when factors are correlated.
• Develop a provably convergent, practical algorithm for tensor completion that works under realistic assumptions, including non-orthogonal and highly correlated factors.
• Demonstrate that the algorithm scales efficiently to high-dimensional tensors (e.g., 1000×1000×1000) with only a tiny fraction of observed entries.
• Show that the algorithm's convergence rate is significantly faster than standard alternating minimization in both synthetic and real-world settings, even under noisy conditions.

Proposed method

• Introduce a new variant of alternating minimization, called Kronecker Completion, which modifies the update rule by incorporating a structured, low-rank approximation of the Hessian to better capture curvature in the tensor space.
• Adapt progress measures from matrix completion to the tensor setting by reweighting the optimization subproblems to account for factor correlation, ensuring stable and fast convergence.
• Use a subsampling strategy during each alternating minimization step, where only a random subset of observed entries is used per iteration, improving stability and convergence speed.
• Maintain three factor matrices (X, Y, Z) of size n×r and update them sequentially by solving small least-squares problems restricted to observed entries, enabling nearly linear time complexity per iteration.
• Formulate the optimization subproblem as minimizing the squared error over observed entries: $\min_{z_1,\dots,z_r} \left\| (T - \sum_{i=1}^r \widehat{x}_i \otimes \widehat{y}_i \otimes z_i) \big|_S \right\|_2^2 $, with updates guided by the new progress measure.
• Integrate a Kronecker structure into the Hessian approximation to improve conditioning and accelerate convergence, especially when factors are correlated.

Experimental results

Research questions

• RQ1 Can we design a tensor completion algorithm with provable linear convergence guarantees that works even when the factors are highly correlated, a common scenario in real-world applications?
• RQ2 Does a modified alternating minimization scheme, informed by matrix completion progress measures, outperform standard alternating minimization in practice and theory?
• RQ3 Can we achieve nearly linear-time complexity for tensor completion while maintaining strong convergence and accuracy, even with sparse observations?
• RQ4 How does the algorithm perform under noisy conditions, and does it achieve near-optimal error scaling with respect to sample size?
• RQ5 Why does standard alternating minimization fail to converge or exhibit counterintuitive behavior (e.g., slower convergence with more data) in correlated settings, and can this be fixed?

Key findings

• Kronecker Completion achieves linear convergence to the true tensor even when the factors are highly correlated, a regime where standard alternating minimization fails to converge or converges extremely slowly.
• The algorithm converges in approximately 30 iterations for 1000×1000×1000 tensors with only 50,000 to 200,000 observed entries, while standard alternating minimization fails to converge to zero error after 1,000 iterations.
• With 10% i.i.d. noise on entries, both Kronecker Completion and standard alternating minimization achieve error below the noise level, and the error decays as the square root of the number of samples, indicating near-optimal performance.
• The sample complexity of Kronecker Completion scales as $n^{3/2}$, consistent with theoretical lower bounds, while standard alternating minimization struggles to scale beyond small tensors and shows pathological behavior with increasing observation counts.
• Subsampling improves the stability and convergence of standard alternating minimization, but the improvement is insufficient to overcome its fundamental instability under factor correlation, which Kronecker Completion resolves.
• In experiments with correlated tensors, Kronecker Completion consistently outperforms standard alternating minimization in convergence speed and final error, even when the latter is run for 10× more iterations.